1431789 - use schema 9 blobs when submitting release blobs

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement)

Description

[[e-mail] (:bhearsum)]

bug 1400016 is adding a new release blob schema that supports delegating What's New Page decisions into it (instead of requiring multiple releases and rules to handle). We should upgrade release blobs to schema 9 after it lands.

We're currently submitting schema 4, but not much has changed that will make the upgrade difficult. We'll have to stop submitting platformVersion, but I think that's it.

This bug may or may not automate the creation of the what's new page parts of the blob -- that often requires late (post go to build) coordination with RelMan, and may not be suitable for automation at this time. Even if we don't do that automatically, upgrading initial submission to schema 9 makes it much easier for a human to come along and do later. Upgrading to schema 9 also allows us to add binary transparency attributes when we're ready to do so.

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/balrog/pull/468

Comment 2

[[e-mail] (:bhearsum)]

Attachment: fix bug in V9 schema validation

Comment 3

[[e-mail] (:bhearsum)]

Attachment: add ReleaseCreatorV9 & ReleaseSubmitterV9

These new classes let us submit V9 blobs to Balrog. In addition to moving detailsURL to the updateLine section, we also need to stop submitting platformVersion from both the top level, and the locale level.

I've done some local tests with this, but I intend to do additional verification with a staging release once the balrogworker and balrog patches for it are in place.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8958588 [details] [diff] [review]
add ReleaseCreatorV9 & ReleaseSubmitterV9

I was wondering why ReleaseCreatorV9.generate_data() accepts an openURL arg but doesn't do anything with it, unlike ReleaseCreatorBase. Suspect I'm missing some part of the bigger picture. Otherwise looks fine to me.

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #4)
> Comment on attachment 8958588 [details] [diff] [review]
> add ReleaseCreatorV9 & ReleaseSubmitterV9
> 
> I was wondering why ReleaseCreatorV9.generate_data() accepts an openURL arg
> but doesn't do anything with it, unlike ReleaseCreatorBase. Suspect I'm
> missing some part of the bigger picture. Otherwise looks fine to me.

Ah, I removed openURL from the body of the method (we don't support it at the top level anymore) but didn't remove it from the constructor. Since we never use automated WNP set-up, I should finish ripping that out from V9 + callers.

Comment 6

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/46a34c5bc36582e9756ba1685e14375d68bacd2c
bug 1431789: fix bug in validation when updateLine is unset in schema 9 blobs (#468). r=nthomas

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8958588 [details] [diff] [review]
add ReleaseCreatorV9 & ReleaseSubmitterV9

(In reply to Ben Hearsum (:bhearsum) from comment #5)
> Ah, I removed openURL from the body of the method (we don't support it at
> the top level anymore) but didn't remove it from the constructor. Since we
> never use automated WNP set-up, I should finish ripping that out from V9 +
> callers.

OK, I'll r+.

Comment 8

[[e-mail] (:bhearsum)]

Attachment: finishing ripping out openURL from V9 classes

Nick - it sounds like you wanted to carry forward the r+, feel free to have another look if that's not the case.

Comment 9

[[e-mail] (:bhearsum)]

I tested this in staging by deleting a recent staging releases' release blob, and rerunning the tasks after aki had deployed the tools+balrogworker patch to the dev workers. As far as I can tell, everything worked fine.
The graph with reruns: https://tools.taskcluster.net/groups/U3BbFLUDRBaUq8vYS5fLCw
The new release blob: https://aus4.stage.mozaws.net/api/v1/releases/Firefox-59.0b17-build12
An update URL receiving the blob: https://aus4.stage.mozaws.net/update/3/Firefox/57.0/20171202185629/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/beta/default/default/default/update.xml

I'll hold off until next week to land this, to stay out of the way of releases.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8959269 [details] [diff] [review]
finishing ripping out openURL from V9 classes

lgtm

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 8959269 [details] [diff] [review]
finishing ripping out openURL from V9 classes

https://hg.mozilla.org/build/tools/rev/ea7ad618089fb84790831878aeedc63985c68ce1

Comment 12

[[e-mail] (:bhearsum)]

Attachment: deploy balrogscript 2.0.1

I believe I also need to manually update the tools checkout on each machine.

Comment 13

[[e-mail] (:bhearsum)]

Attachment: increase version bump

Comment 14

[Aki Sasaki (not active)]

All balrog tasks for Firefox 60.0b5 are failing with errors like

2018-03-19 22:54:08,750 - ERROR - Caught HTTPError: {
  "detail": "Release name: None not found",
  "status": 404,
  "title": "Not Found",
  "type": "about:blank"
}

requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://aus4-admin.mozilla.org/api/releases/Firefox-60.0b5-build1/builds/WINNT_x86-msvc/en-US

Comment 15

[Aki Sasaki (not active)]

Pushed a backout for release bustage: https://hg.mozilla.org/build/puppet/rev/aeccffd032ba3bc65fe599549e9baea46a8b9e19

16:12 <•nthomas> yeah, it's a KeyError at https://github.com/mozilla/balrog/commit/46a34c5bc36582e9756ba1685e14375d68bacd2c#diff-1a7ea9e25a2f1a8c15c06fd95243142aR1020

Looks like that block needs to be in an `if` statement, otherwise we look for None["for"] a few lines down.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Deploying Balrog 2.48 with the changes which add .get()s would also work (46a34c5 & 9087a3c).

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Oops, I meant that deploying Balrog 2.48 is needed before switching to v9 schema.

Comment 18

[[e-mail] (:bhearsum)]

The Balrog changes are now in production - I'll look into rolling out the balrogworker changes again tomorrow.