1346752 - investigate if balrog workers tasks should depend on beetmover-checksums tasks in the nightly graphs

Status: NEW

(Release Engineering :: Release Automation, defect, P2)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

:jlorenzo raised an interesting point this morning that I think we should discuss. When the nightly builds are done, the beetmover jobs (at least for the multi/en-US locale) transfer the artifacts to S3, to both the `dated` and `latest` locations. As soon as that happens, there are two categories of jobs that get unlocked:

a) balrog jobs which take the manifest.json generated by beetmoverworker and submit all that information to Balrog

b) checksums jobs (both signing and beetmover) that take the checksum file generated by initial beetmoverworker job - sign it, following which another set of beetmover jobs transfer that checksums artifact to their location in S3 (both dated and latest locations).

But this morning, for some reason (could be high load or something else, not sure yet) signing jobs were pending, hence lots of checksums-signing and l10n build signing jobs were pending. In this case, if the main en-US/mobile signing build is done, it unlocks the balrog job which goes green right away while the checksums signing + beetmover checksums are still pending.  This means that we end up with something like the following on S3:

Dir 	.. 		
Dir 	en-US/ 		
File 	bouncer.apk 	308K 	13-Mar-2017 10:13
File 	fennec-54.0a2.multi.android-arm.apk 	39M 	                13-Mar-2017 10:13
File 	fennec-54.0a2.multi.android-arm.checksums 	9K 	        12-Mar-2017 10:00
File 	fennec-54.0a2.multi.android-arm.checksums.asc 	836 	        12-Mar-2017 10:00
.....
.....
File 	fennec-54.0a2.multi.android-arm.common.tests.zip 	75M 	13-Mar-2017 10:13
File 	fennec-54.0a2.multi.android-arm.web-platform.tests.zip 	45M 	13-Mar-2017 10:13
File 	fennec-54.0a2.multi.android-arm.xpcshell.tests.zip 	11M 	13-Mar-2017 10:13


So from a developer standpoint, within the `latest` S3 location - the nightly build is out and served via Balrog but its corresponding  checksums still dates from within the previous day. 

I see two options here:
 
a) Leave it as it is.

plus: nightlies are published fast and we add no other dependency
minus: in the small (could be larger if we have signing servers down or whatever) time frame until the checksums get uploaded as well, we have the aforementioned mismatch between current nightly and the checksums.

b) chain balrog jobs to the checkusms signing

plus: we get rid of the situation above so that when we actuall serve the nightlies, we know for sure that all the bits are in the proper place
minus: the added dependency adds some delay to the total amount of time till we serve the nightlies, as balrog jobs needs to wait for all the signing jobs and checksums jobs to complete before it runs.

Question: what should we do in this case?

Comment 1

[Félix Saparelli [:passcod]]

As a maintainer of an alternative package for firefox on Arch Linux, we depend on the checksums file matching the available nightly to check integrity. In option a), you mention that the time frame for the mismatch is "small". That's not really the case: more often than not, my own attempts and my users' attempts to install result in checksum mismatches. That's been going on, on and off, for at least a week (I only got time to track this down today).

Currently, we're disabling checksums, but obviously that's not a great workaround.

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Félix Saparelli [:passcod] from comment #1)
> As a maintainer of an alternative package for firefox on Arch Linux, we
> depend on the checksums file matching the available nightly to check
> integrity. In option a), you mention that the time frame for the mismatch is
> "small". That's not really the case: more often than not, my own attempts
> and my users' attempts to install result in checksum mismatches. That's been
> going on, on and off, for at least a week (I only got time to track this
> down today).
> 
> Currently, we're disabling checksums, but obviously that's not a great
> workaround.

@passcod Thanks for your input - disabling those checksums is indeed not the desired workaround. I'll discuss this today with the rest of the team to see what's the best solution here.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

We talked about this today and we're to chain the balrog tasks to the beetmover-checksums jobs so that we avoid the race condition there from now on.

ETA: by the end of week.

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Sorry for delays, been caught traveling towards the end of last week.
Am picking this up today with ETA set for EOW.

Comment 5

[Jan Tojnar]

Any progress on this? I got the checksum mismatch again today.

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Jan Tojnar from comment #5)
> Any progress on this? I got the checksum mismatch again today.

Sorry for the delay with this. Other higher-priority stuff landed on our plate recently and this bug somehow slipped. Mea culpa for that. I'll have a look again at this.

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Grabbed this bug the other day and did more digging. Status update:

1. Currently the balrog job depends on all beetmover pieces:

  - beetmover
  - beetmover-l10n
  - beetmover-repackage
  - beetmover-repackage-l10n

2. beetmover-checksums kind depends on checksums-signing which at its turn depends on the same kinds:

  - beetmover
  - beetmover-l10n
  - beetmover-repackage
  - beetmover-repackage-l10n


So technically, we can:
a) chain balrog directly to beetmover-checksums
b) make sure we walk on deps to grab the public/manifest.json that balrog currently gets from beetmover. This can be done by just walking to beetmover-checksums dependencies which currently are beetmover + checksumssigning jobs. 

Caveat:
* balrog currently filters out `nightly` + `signed` attributes whereas beetmover-checksums filters only `nightly`. If I add `signed` as well to this new dependency model, there's no balrog jobs created. So I can either add `signed` to checksums + beetmover-checksums jobs or .. filter that out somehow. I need to investigate this
* we're currently rolling out macosx nightlies to central so this week may not be the best one to change dependencies. 

Will try to finish patches though and get everything all set for next week.

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

Not touching this anymore until the full tcmigration is completed (mac/win). There's a large set of cleanup that we're aiming after that so this bug should fall into that category as well.

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

TODO: revert https://github.com/mozilla-releng/mozapkpublisher/pull/42 once this bug is fixed.