1398805 - Add checksums builder in-tree

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Description

[Chris AtLee [:catlee]]

Perhaps this gets done as part of beetmover work in bug 1332339?

Comment 1

[Aki Sasaki (not active)]

If possible, let's converge on nightly - so support all checksum file types in both release+nightly.

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Kim and I talked about how to tackle this last week. I'm having some seconds thoughts. I think it might be worth going with what Aki said in the first place. I think I can tweak beetmover to add those missing *.beet files so that we can reuse the existing script via BBB. This way we can tackle rewriting the checksums in the cleanup/future.

I'm tweaking the beetmoverscript part to see if this is feasible. Will follow-up here with a PR.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Beetmoverscript is now updated to encompass checksums as well. It's slightly different than the old (mozharnes) beetmoverscript approach, but in a way similar.

Idea is to:
1. Cocentrate all the checkscums information in form of .beet files under beetmover-checksums (e.g. [1]) 
2. Enable BBB checksums builder (this will gather them all and build up the SHA256/512 SUMS + SUMMARY).

1) is done - see beetmover-checksums green builds from this graph[2]. 
Adding 2) will be slightly trickier as we need to make sure checksums is the last task to run in the graph. We'll need some dummy tasks there too + all the others as deps but other than that, should be good.

[1]: http://ftp.stage.mozaws.net/pub/firefox/candidates/58.0b12-candidates/build6/beetmover-checksums/
[2]: https://tools.taskcluster.net/push-inspector/#/L-a-hZzMR4K_WvAQNXH4_A

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Stealing this off Kim's plate as I manage to have a look at it.
Pushed part-1 https://hg.mozilla.org/projects/maple/rev/bfe8fd197a2e97eb3cbd40ad9e2b03e4b4493faf

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Checksums builder

Asking for some feedback, in case I've missed something. Seems to work properly on my local graphs, I'll trigger a staging release to see how it behaves.

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 8935509 [details] [diff] [review]
Checksums builder

Ship it [on maple] and test it :)

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

So the job[1] failed with some good logs[2]. Sounds like I accidentally moved the signature files (.asc) as well as .beet files which is obviously breaking one of the reading-content functions[3]. Solution is to simply keep .asc files as they are since they will be ignored by the checksums script[4].

[1]: https://tools.taskcluster.net/groups/bn9Ab9njQUugb5rKcemuvg/tasks/aVp3vGlARK2neAlMOJsHxw/details
[2]: https://archive.mozilla.org/pub/firefox/tinderbox-builds/maple-noarch/release-maple-firefox_chcksms-bm71-build1-build0.txt.gz
[3]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/mozharness/mozilla/checksums.py#l6
[4]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/scripts/release/generate-checksums.py#l216

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

http://ftp.stage.mozaws.net/pub/firefox/candidates/58.0b14-candidates/build5/ worked.
Will add a trello card to make sure there are no missing (important) files in the checksums.

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

Everything looks as expected, except source file which is not included as it's not part of the beetmover-checksums chunk. We can add it later, but in the cleanup section, there's a trello card for that.