1315415 - docker-image sha doesn't match

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Aki Sasaki (not active)]

For Chain of Trust verification, I need to make sure that the docker image we use is the same as the docker image we built in the appropriate docker-image build.

For instance:
Build HvuyKlX0TcSCwEBsOgOqFQ uses the docker image from AwZX1zIzT1WJ984RCzEDew.

The build's chain of trust has an environment.imageHash of sha256:a6d5cdd4545d8ed6707dc0b5141570b1e5d88492f0e54738f464a7ba00467eee , which lets us verify that the correct image is running.  The task.payload.image specifies a path of "public/image.tar", and the taskId of AwZX1zIzT1WJ984RCzEDew, which is as expected.

The docker-image task's chain of trust artifact has no mention of the above imageHash.  I suspect this is because the docker-image's cot specifies the sha of public/image.tar, and the cot.environment.imageHash specifes the sha of the image once it's extracted from the tarball.

I also use cot.environment.imageHash to verify the image hashes of the decision and docker-image tasks, because they download from docker hub.  So we don't want to remove that.

We need either need
a) something in the build's task or chainOfTrust that lets us know what the public/image.tar sha is, or
b) something in the docker-image's artifacts or chainOfTrust that lets us know what the docker image sha is.

Comment 1

[Greg Arndt [:garndt]]

You're absolutely right.  The CoT artifact created by the image task references a sha256 hash of the image.tar artifact that was uploaded.  This is the default behavior of any artifact uploaded by a worker and captured in a CoT artifact.  However, tasks that use that image are logging the image hash as reported by docker once the image has been imported, which of course would be different.  You can see that the image hash in the CoT of the builder matches the hash that docker reports when importing the image.

Would the acceptable solution here be the same sha256 that the image builder task reports ( the hash of image.tar artifact), is the one that the builder task reports as well in the CoT?  In that case, I think that it would be simple enough.

Comment 2

[Aki Sasaki (not active)]

That would work for my purposes.  If the docker hub (decision, docker-image) environment.imageHash is the docker image's hash, and any job that uses a an image built from a docker-image task reports the image.tar hash, I think that's an acceptable solution, although it may be confusing to someone unfamiliar with the convention.  Maybe we just document it?

An alternative might be to add another environment entry for image.tar if applicable, e.g. environment:

  {"public/image.tar": "sha256:...", ...}

Those both work for me, as does an artifact on the docker-image side that reports the built docker image sha.

Comment 3

[Greg Arndt [:garndt]]

I could see where the confusion would happen.  environment.imageHash could be an overloaded term, where it mean the actual context hash that docker calculates, or the hash of the image tarball stored as an s3 artifact.

I would suggest that we make it more explicit and have a hash that docker reports, and a hash of the artifact if an artifact task image was used.  We would have environment.imageArtifactHash = abc.  I don't have strong opinions on where to stick it.

Comment 4

[Aki Sasaki (not active)]

Sounds like you're agreeing with two different environment items... that works for me!
I'm currently working around this with a warning.  We'll need to fix this bug (and roll out the appropriate scriptworker change to check the new var and fail instead of warn) to have end-to-end chain of trust verification.

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/taskcluster/docker-worker

https://github.com/taskcluster/docker-worker/commit/3ee9c2ec38e837eb658d0b5c43eeab6e61340c0f
Bug 1315415 - Add hash of task image artifact

https://github.com/taskcluster/docker-worker/commit/f41b32b6ba9d718be36ea32e2a7b0df9778e97e3
Merge pull request #262 from gregarndt/docker_image_hash

Bug 1315415 - Add hash of task image artifact to chain of trust certi…

Comment 6

[Aki Sasaki (not active)]

https://github.com/mozilla-releng/scriptworker/pull/33 landed.  I'm able to verify the docker-image sha via verify_cot locally; scriptworker instances will be able to with the next scriptworker release.

Thanks!

Comment 7

[Greg Arndt [:garndt]]

awesome \o/