1380829 - Add binary transparency task to releasetasks

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement, P1)

Description

[Brandon Tang [:btang]]

- Investigate templates (specifically checksums, as well as a taskcluster-native task) to learn about releasetasks and find definition differences
- Determine what to migrate over from manually configured binary transparency tasks
- Determine how to test generation of task graph locally
- Be able to run releasetasks tests locally
- Look at releasetask test files to understand the structure of releasetasks and its test structure
- Save a graph to file from a test for manual inspection
- Add a test for binary transparency

Comment 1

[Brandon Tang [:btang]]

Attachment: bug1380829-buildbot-configs-v1.diff

- added binary_transparency_enabled for mozilla-beta

Comment 2

[Brandon Tang [:btang]]

Attachment: bug1380829-tools-v1.diff

- added binary_transparency_enabled to: release runner kwargs, (manual generative) release runner, and sanity kwargs checker
- also added into parameters of make_task_graph_strict_kwargs function

Comment 3

[Brandon Tang [:btang]]

Attachment: bug1380829-tools-v2.diff

- updated binary_transparency_enabled in release-runner to default to False if it doesn't exist instead of hardcoding True

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Commented in the bug for the PR as well - https://github.com/mozilla-releng/releasetasks/pull/254

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Esr45 is dead, cleanup configs in releasetasks.

Hijacking this bug for a second to do a cleanup since we're here. Let's kill esr45 with fire!

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8890136 [details] [review]
Esr45 is dead, cleanup configs in releasetasks.

Approved by :nthomas in PR.

Comment 7

[Brandon Tang [:btang]]

Attachment: bug1380829-tools-v3.diff

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8889965 [details] [diff] [review]
bug1380829-buildbot-configs-v1.diff

https://hg.mozilla.org/build/buildbot-configs/rev/b703cd920843
https://hg.mozilla.org/build/buildbot-configs/rev/b04e09ed9afa

Comment 9

[Brandon Tang [:btang]]

Comment on attachment 8890572 [details] [diff] [review]
bug1380829-tools-v3.diff

https://hg.mozilla.org/build/tools/rev/f8a63f49524e9c70cd1d2c4aefa9042d8fdec024

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Updated bm85's releaserunner from rev fbc86f515ef1 to f8a63f49524e, and restarted releaserunner.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

releasetasks merged - https://github.com/mozilla-releng/releasetasks/commit/546ea6d41045b78540884dff0961321c5fb4bf13

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

For some reason we didn't get a binary transparency task in the graph for 55.0b13, https://tools.taskcluster.net/groups/SCJmm1VGS4ip3XEQ-BaX0g, so we'll have to figure out why that is.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Ah, I should have done 'git pull' in bm85:~cltbld/releasetasks/ when deploying. Just updated from 06d13f5653c5be3ab4af3933e30da0911240ca06 to 546ea6d41045b78540884dff0961321c5fb4bf13.

Lets close this once we successfully generate a cert.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Enable binary transparency on jamun

bug 1385134 plans a staging 56.0b1 we should enable BT on the staging branch (jamun). I expect the job will fail because of the hardcoded archive.m.o at https://github.com/mozilla-releng/releasetasks/blob/546ea6d41045b78540884dff0961321c5fb4bf13/releasetasks/templates/desktop/binary_transparency.yml.tmpl#L28. I think that's fine from the point of view of testing the graph generation and making sure the job runs.

Comment 15

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8891131 [details] [diff] [review]
[buildbot-configs] Enable binary transparency on jamun

Lgtm!

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8891131 [details] [diff] [review]
[buildbot-configs] Enable binary transparency on jamun

https://hg.mozilla.org/build/buildbot-configs/rev/075e8cd5adc1844adf94b4a8d5ef057d6584b7e2
https://hg.mozilla.org/build/buildbot-configs/rev/534f8494d95bf8e7730f75c44c9cf62439474d51

Comment 17

[Mihai Tabara [:mtabara]⌚️GMT]

We had a staging release going on on Friday with this[1] graph. (central => jamun) I couldn't find binary transparency job there but:
a) there's lots of other tasks missing so I suspect there might be some misconfiguration somewhere
b) did we do a reconfig after https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/project_branches.py#l201 ? if not, should have we?

[1]: https://tools.taskcluster.net/push-inspector/#/d9RbMcpIS-2aRxImMKa8cA

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Looks like I didn't deploy to the dev releaserunner for the changes here. So on bm83 I 
* updated ~cltbld/releasetasks/ from e9c41655ba706751c9e59528babeb2e94614bd1c to 546ea6d41045b78540884dff0961321c5fb4bf13
* updated /builds/releaserunner/tools from 077a4adc87dc to f686c083bf49  (to pick up f8a63f49524e from this bug)
* restarted releaserunner using supervisorctl

There is a 'checksums builder' task on d9RbMcpIS-2aRxImMKa8cA, so I'm hopeful that the next staging run will have a binary transparency worker.

Comment 19

[Mihai Tabara [:mtabara]⌚️GMT]

The current staging release 56.0b1-build17 picked it up - the job is now created[1] \o/
Let's see how it goes.

[1]: https://tools.taskcluster.net/groups/PoPbnpb5Q1iRrbjmsFTV9A/tasks/_3z49ZMyTImqsY3rX6vkkg/details

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Woo! It's failing in the expected way (see comment #14), so lets be positive and close this as FIXED again.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

We got a job for DevEdition 55.0b14 too, nice and green - https://tools.taskcluster.net/groups/td2Qd2qfTDqga8SUgyIAMg/tasks/uQ1QaUsqTUeFRJ-sopaffw/details