990689 - don't wait for all platforms to finish before running update verify

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect, P1)

Description

[[e-mail] (:bhearsum)]

Update verify currently runs after the "updates" builder because we have to wait for all platforms to finish their build/repacks before we can create and push snippets.

Once Balrog is being used for betas/releases we'll have all the data we need for update verify in Balrog as each platform finishes. Given that, there's really no reason to artificially delay running it until later. This won't really help with end to end time, as Windows repacks + update verify are still critical path, but it would help in situations where Linux or Mac becomes the long pole because of issues, and should help take some load off of FTP, too.

We'd need a few tweaks to do this. At least the following:
* Move initial balrog blob submission/test channel updates to the start of the release process
* Get rid of the updates builder complete
* Generate update verify configs elsewhere (perhaps in each of the update verify builders -- there's no reason we have to keep a checked in version now that they're generated based on the patcher configs).
* Adjust schedulers roughly as follows:
tag -> {initial balrog,linux build,    linux64 build,    win32 build,    mac build}
                           |                 |                 |              |
                       linux repacks   linux64 repacks   win32 repacks   mac repacks
                           |                 |                 |              |
                       linux uv        linux64 uv        win32 uv        mac uv
                           \-----------------\-------|---------/--------------/
                                                     |
                                              ready for release

check permissions, antivirus, etc. would still trigger off of the "deliverables ready" scheduler and feed into "ready for release" (in addition to the update verifies in the graph above)

Comment 1

[[e-mail] (:bhearsum)]

There's probably still an additional blocker to this - we won't be getting rid of the updates builder completely as part of bug 933414.

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #1)
> There's probably still an additional blocker to this - we won't be getting
> rid of the updates builder completely as part of bug 933414.

The additional blocker is more or less "move top level balrog submission to start of release process". Basically, we need to split https://github.com/mozilla/build-tools/blob/master/scripts/updates/balrog-release-pusher.py into two scripts: one that runs ReleaseCreator, and one that runs ReleasePusher.

If we do that, the rest of this bug is just adjusting the update verify schedulers to fire after each platform's repack_complete builder finishes.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

What would run the ReleasePusher part ? An cut-down updates builder ? :-P  We could have separate platform rules, but instead of splitting we could just run it all early. ie point test channels at the new release straight away, and don't mind no updates until repack_complete comes in.

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #3)
> What would run the ReleasePusher part ? An cut-down updates builder ? :-P

Depends on where we want to do update verify config bumping. We could do it with ReleasePusher or ReleaseCreator. Either way, it might be a good time to move some of this to Mozharness.
 
> We could have separate platform rules, but instead of splitting we could
> just run it all early. ie point test channels at the new release straight
> away, and don't mind no updates until repack_complete comes in.

Good idea.

I also realized that QA can probably start their update tests earlier, too. We should consult with them if/when we do this.

Comment 5

[Rail Aliiev [:rail]]

This should be easy to do in relpro.

Comment 6

[Aki Sasaki (not active)]

We currently block on all platforms in TC relpro, but only b/c we have an all-platforms set of beetmover dummy tasks.
If we split the beetmover dummy tasks by build_platform, and only depend on the dummy tasks that match our build_platform(s), then this should be solvable in-tree.

Comment 7

[[e-mail] (:bhearsum)]

I'm looking into this as part of tcmigration cleanup work.

Comment 8

[Phabricator Automation]

Attachment: per platform beetmover + balrog dummies

Comment 9

[Phabricator Automation]

Comment on attachment 8961896 [details]
per platform beetmover + balrog dummies

Aki Sasaki [:aki] has approved the revision.

https://phabricator.services.mozilla.com/D799

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/39eed777c0c4
don't wait for all platforms to finish before running update verify. r=aki

Comment 11

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/39eed777c0c4