1192674 - Avoid push races for update jobs in release builds, and update verify failures on beta channel

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom] Rejig scheduling of updates_beta

On x.0 release builds we have full sets of updates and update_verify (uv) jobs for both the beta and the release channels. The updates jobs are scheduled at the same time, and we don't have any retry on pushing the patcher & uv configs into the tools repo, so there's often a push race and one of them loses. 

If the beta_updates job wins, we get failures for x.0 --> y.0 updates where there's a partial. This is sorta a Balrog bug - there's a partial declared at the locale level in the blob, but the matching entry in the fileUrl block is missing. The release_updates job is responsible for adding the config to fileUrl block, and that failed earlier. We end up with no update offered. This also breaks the beta_final_verification job.

This patch works around this by running the release_updates job first, then kicking off the beta_updates job at the same time as the release update verify. I haven't worried about staging or update verify disabled when writing this.

The diff in a master dump (with all the release branches enabled) looks like this:
--- all.before	2015-08-10 09:32:12.000000000 +1200
+++ all.after2	2015-08-10 12:02:23.000000000 +1200
@@ -21807,8 +21807,7 @@
  'upstreamBuilders': ['release-mozilla-release-firefox_push_to_mirrors',
                       'release-mozilla-release-firefox_release_updates']}
 <buildbotcustom.scheduler.AggregatingScheduler> {'branch': 'releases/mozilla-release',
- 'builderNames': ['release-mozilla-release-firefox_release_updates',
-                  'release-mozilla-release-firefox_beta_updates'],
+ 'builderNames': ['release-mozilla-release-firefox_release_updates'],
  'enable_service': False,
  'name': 'release-mozilla-release-firefox_signing_done',
  'okResults': (0, 1),
@@ -22067,7 +22066,8 @@
                   'release-mozilla-release-win32_update_verify_release_9/12',
                   'release-mozilla-release-win32_update_verify_release_10/12',
                   'release-mozilla-release-win32_update_verify_release_11/12',
-                  'release-mozilla-release-win32_update_verify_release_12/12'],
+                  'release-mozilla-release-win32_update_verify_release_12/12',
+                  'release-mozilla-release-firefox_beta_updates'],
  'enable_service': False,
  'name': 'release-mozilla-release-release_updates_done',
  'okResults': (0, 1),

The beta update verify still depend on the beta_updates builder.

Comment 1

[[e-mail] (:bhearsum)]

Comment on attachment 8645514 [details] [diff] [review]
[buildbotcustom] Rejig scheduling of updates_beta

Review of attachment 8645514 [details] [diff] [review]:
-----------------------------------------------------------------

Clever solution! We should still fix the balrog submitter issue at some point (I imagine the graph will change quite a bit in release promotion), but this seems like a fine workaround for now.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Planning to land this once 40.0 and 41.0b1 ship.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8645514 [details] [diff] [review]
[buildbotcustom] Rejig scheduling of updates_beta

https://hg.mozilla.org/build/buildbotcustom/rev/2feeea61042b

Comment 4

[Kim Moir [:kmoir] ET]

In production