1038875 - add builder that pushes releases live in balrog

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[[e-mail] (:bhearsum)]

We'll be shipping Balrog to the Beta channel soon. Rather than make people use the Balrog UI (whose rules.html page is still extremely slow), we should get a Buildbot builder to do it through the API. It should be similar to the postrelease builder in that it's not hooked up to any schedulers.

I think it would also be good to require a magic keyword to be present in the properties to avoid someone accidentally shipping hitting "force build" errantly.

Comment 1

[[e-mail] (:bhearsum)]

I'm planning to give the reviews for this to Rail since it's all client side code, but do you have any thoughts or reasons why we shouldn't do this, Nick? I don't think we've ever talked about it...

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I feel slightly paranoid about this, but then remind myself it will remove some chances for human frailty to creep in. The magic keyword is a good idea for preventing oopsies.

Ideally, after ship-it is tracking status it can ensure that we have completed enough checks (eg update verify and final verification) before enabling the button to push updates.

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #2)
> I feel slightly paranoid about this, but then remind myself it will remove
> some chances for human frailty to creep in. The magic keyword is a good idea
> for preventing oopsies.
> 
> Ideally, after ship-it is tracking status it can ensure that we have
> completed enough checks (eg update verify and final verification) before
> enabling the button to push updates.

Yeah, that's my thinking too. I'd go so far as to say that the server side should protect against somebody trying to forge that button press too.

I'll forge ahead here, and we can feel paranoid together!

Comment 4

[[e-mail] (:bhearsum)]

Attachment: boring buildbot configs patch

This just adds releaseChannelRuleIds to release configs. It's empty for everything except beta for now. We may need additional IDs at some point because Firefox and MetroFirefox have different rules. I don't think we need it yet though, because we haven't shipped Metro to Beta.

Comment 5

[[e-mail] (:bhearsum)]

Attachment: buildbotcustom work

One thing of note here is removing "buildprops.json" as a magic string.

I also didn't bother adding a .sh wrapper for this - do you think that's okay? It seemed largely unnecessary to me.

Comment 6

[[e-mail] (:bhearsum)]

Attachment: script to ship releases

This script is pretty much a pared down version of balrog-release-pusher.py. It doesn't need to use the ReleaseCreator class, and therefore doesn't need as much information from the release config.

Comment 7

[[e-mail] (:bhearsum)]

Here's a couple of sample runs, too. This one was without the magic property: http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-mozilla-beta-update_shipping/builds/8

This one was with it: http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-mozilla-beta-update_shipping/builds/9

They updated https://aus4-admin-dev.allizom.org/rules.html

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 8458006 [details] [diff] [review]
buildbotcustom work

Review of attachment 8458006 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +1233,5 @@
> +                '-r', releaseConfigFile,
> +                '-a', balrog_api_root,
> +                '-u', balrog_username,
> +                '-c', 'oauth.txt',
> +                '-p', 'buildprops.json', 

Can you remove the trailing space when you land?

Comment 9

[[e-mail] (:bhearsum)]

I landed things on default. It's likely that they'll go into production before 33.0b1, so we can test it there with that!

Comment 10

[[e-mail] (:bhearsum)]

I updated docs:
https://wiki.mozilla.org/index.php?title=Release%3ARelease_Automation_on_Mercurial%3AUpdates_through_Shipping&diff=998013&oldid=998011
https://wiki.mozilla.org/index.php?title=Releases%2FRelEngChecklist&diff=998017&oldid=997088

Since this involves a change to release procedure, I'll also send mail in a bit.

Comment 11

[Aki Sasaki (not active)]

In production.

Comment 12

[[e-mail] (:bhearsum)]

This worked perfectly fine in 32.0b1!