1674131 - ensure transparency log is up to date as part of release promotion

Status: NEW

(Release Engineering :: Release Automation, enhancement)

Description

[[e-mail] (:bhearsum)]

The only thing we should have to do here is run ./tl verify $URL $LOCAL_FILE against all of the files we publish to /releases. This means it has to be done after we push to CDNs - probably at the same time as final verify. We should make sure the $LOCAL_FILE is fetched from somewhere trusted -- ideally this would rely on upstream task artifacts, and be chain of trust enabled, but I'm not sure how practical that will be - I will dig into it more later.

There's a maintained Docker image that we can probably use for this, rather than building our own: https://quay.io/repository/transparencylog/tl

For more background on transparency log, see https://github.com/transparencylog/tl

Comment 1

[[e-mail] (:bhearsum)]

I dug into this a bit more today. I think our best path forward in the short term is to simply add a docker worker task that runs in either the push or ship phase of a release (depending on whether or not we think we'll ever overwrite artifacts), and runs tl verify on all the artifacts in the a releases directory. This short term method means we won't be verifying CoT - but it will be much quicker to bootstrap.

Down the line, if we want to verify CoT we can switch from docker worker to a scriptworker.

I've got something locally that's about half done, that uses a SHA256SUMS file to generate a bunch tl verify commands, and I'm currently working on getting it scheduled in Taskcluster. The script will need some tweaks after https://github.com/transparencylog/tl/issues/27 is done, so we can verify based on the hashes in SHA256SUMS instead of needing files on disk.

Comment 2

[[e-mail] (:bhearsum)]

Attachment: Bug 1674131: Add ftp -> bouncer platform mapping

Comment 3

[[e-mail] (:bhearsum)]

Attachment: Bug 1674131: Add wrapper script to run 'tl verify'

Depends on D98031

Comment 4

[[e-mail] (:bhearsum)]

Attachment: Bug 1674131: add transparency log and python mozrelease directories to update verify sparse profile

Depends on D98032

Comment 5

[[e-mail] (:bhearsum)]

Attachment: Bug 1674131: Add transparency log task to push phase of firefox & devedition releases

Depends on D98033

Comment 6

[[e-mail] (:bhearsum)]

My patch stack is not quite ready for review, but it's pretty far along. The most notable things missing are:

• Testing to make sure the new transparency log tasks work
• https://github.com/transparencylog/tl/issues/27 being fixed, or me implementing a workaround of download files before verifying them