1320823 - Lock external docker images with content hash

Status: RESOLVED FIXED

(Taskcluster Graveyard :: Docker Images, defect)

Description

[Jonas Finnemann Jensen (:jonasfj)]

Whenever, we reference external images we should include image hash, even if we
don't really validate the images, locking the content hash ensures they aren't
used as an attack vector.

We reference external images in two cases:
 A) Images that are manually built like image_builder
 B) Source images that we reference in in FROM clause in the Dockerfile

A) Can be solved by adding an additional file: HASH, next to the existing REGISTRY and VERSION files. A better options might be to replace the 3 files
with a single file called IMAGE_REFERENCE or something like that.

B) Can be solved by including the image hash in the FROM clause.

Note: (A) would also make verification using COT artifacts easier.
As there is no need to compare the image hash with a list of approved hashes,
if the task definition signed by the decision task in task-graph.json contains
the same image:tag:hash for the given task.
(Validating against a set of approved hashes is still solid hardening).

Comment 1

[Aki Sasaki (not active)]

Anyone with the right scopes can create any arbitrary decision task, so having a hash in the task definition doesn't guarantee anything.

Comment 2

[Gregory Szorc [:gps]]

In the version-control-tools repo, we introduced custom Dockerfile syntax for managing the base images. e.g. https://hg.mozilla.org/hgcustom/version-control-tools/file/818e6844bb8a/testing/docker/builder-ansible-centos7/Dockerfile#l5.

When building images from Dockerfiles, we preprocess the Dockerfile looking for that syntax. If it is found, we download the referenced URL, verify the hash, import the image, then rewrite the Dockerfile as part of the context tar uploaded to Docker to reference the just-imported image. That code lives at https://hg.mozilla.org/hgcustom/version-control-tools/file/115809a6a8e1/testing/vcttesting/docker.py#l431 and https://hg.mozilla.org/hgcustom/version-control-tools/file/115809a6a8e1/testing/vcttesting/docker.py#l325.

Aside from pinning hashes of downloaded images, we can also host images on any URL we want. Yay for not having to run a registry and introducing another point of failure!

The comments in the linked code about Docker's insecurity were written a while ago. I believe the comment about Docker not verifying the hash before processing the image is still accurate.

Comment 3

[Aki Sasaki (not active)]

Correct me if any of these assumptions are incorrect:

* we need the docker image for the decision task before we can run mach
* if the docker-worker had the ability to read a custom syntax and verify contents before running, we could pin a docker sha in the decision task definition
* however, anyone with the right scopes can submit any arbitrary decision task definition, meaning the pinning is not secure

That last point is what I'm mainly concerned about.  I would love to find a way around that that doesn't involve an allowlist, but I haven't figured it out yet.

Comment 4

[Jonas Finnemann Jensen (:jonasfj)]

Attachment: Bug 1320823 - Reference prebuilt docker images by HASH.

This will make harder to falsify a decision task.
Notably our validation code only needs to verify that the definition of the
decision task as given here matches what is used in the task definition in
the Chain-Of-Trust artifact, in order to prove that the decision task is
a result of what ran in the tree.

Review commit: https://reviewboard.mozilla.org/r/98954/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/98954/

Comment 5

[Jonas Finnemann Jensen (:jonasfj)]

@aki, when you get back, take a look at:
  https://reviewboard.mozilla.org/r/98952/diff/1#index_header
The idea is that now we don't need a whitelist, we can just verify the task definition from the COT artifact
against the contents .taskcluster.yml file.

For this to be feasible we probably have to undertake a few more simplifications, otherwise, we have to do
a lot of magic parameter substitution for .taskcluster.yml

Comment 6

[Jonas Finnemann Jensen (:jonasfj)]

Comment on attachment 8819115 [details]
Bug 1320823 - Reference prebuilt docker images by HASH.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/98954/diff/1-2/

Comment 7

[Jonas Finnemann Jensen (:jonasfj)]

Filed as two blocking bugs, so I won't get conflicts in mozreview.

Comment 8

[Jonas Finnemann Jensen (:jonasfj)]

Seems this is all fixed...