1461756 - (fx-hermetic-ci) [meta] Hermetic Firefox CI tasks

Status: NEW

(Firefox Build System :: Task Configuration, task)

Description

[Gregory Szorc [:gps]]

Currently, various Taskcluster tasks in Firefox CI touch external services. For example:

* Building Docker images requires fetching the base image from Docker Hub
* Building Docker images requires installing packages from a distro's package repository (e.g. archive.debian.org, Ubuntu and CentOS equivalents)
* Building toolchains requires fetching source archives from random servers (e.g. ftp.gnu.org)
* Random tasks also touch github.com, PyPI, NPM and other servers

If a task has a run-time dependency on an external service and that service goes down or changes from under us (say it deletes a file we depend on), it may break parts of Firefox CI. In the worst case, this could prevent us from expediently shipping Firefox, including a chemspill. That's obviously not good.

A mitigation to this problem is to have Firefix tasks rely on as few external resources as possible. Ideally, a task's inputs should only be artifacts from other tasks. For inputs that are initially only available on a remote resource, we would have special tasks that "fetch" these inputs and re-expose them as task artifacts for consumption into dependent tasks.

Such tasks are "hermetic:" they are sealed and their inputs are well-defined. They aren't prone to intermittent availability of remote services (beyond the Taskcluster platform) and aren't prone to behavior of remote services changing over time (including content changing or being removed).

I'm filing this bug as a tracker for all "make Firefox CI tasks hermetic" issues.

Comment 1

[Gregory Szorc [:gps]]

Bloating scope to include all of CI.