1347212 - improve release automation publishing of releases

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement, P2)

Description

[[e-mail] (:bhearsum)]

Currently, release automation is responsible for updating the live release rules (that is, the ones specified in "publish_rules" in files like https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/releases/updates_firefox_release.py#43) when we ship. There's a few issues with it:
- Humans have to manually adjust the backgroundRate to 25% first (for release channel)
- Humans need to remember to set fallbackMapping (for release channel)
- Humans have to be around to it at all!
- The current automation will break when Required Signoffs are added

We can fix all of these things by updating the existing automation to:
- Schedule a change when we push to cdntest, instead of directly manipulating the rule when we ship
-- Rail suggested that we may be able to grab the desired ship time from Ship It, and set that as the "when" for the change
- Make the automation set backgroundRate to 25%, (release channel only)
- Make the automation set the current mapping to fallback mapping, so that users on n-2 or older versions can still update somewhere while updates are throttled (release channel only)

Comment 1

[[e-mail] (:bhearsum)]

Rail, does ship it already contain the desired ship time?

Comment 2

[Rail Aliiev [:rail]]

No, but it shouldn't be too hard to add this.

Comment 3

[[e-mail] (:bhearsum)]

Rail and I are going to talk more about Ship It integration next week. If we don't do that, other options include:
* Try to guess in automation, but that seems kindof scary
* Guess in automation and then add a manual step to set the publish time correctly by hand
* Schedule the change right at publish time - but that would mean that relman would give the go, we would unblock the human decision task, and then they'd have to go signoff in balrog
* Remove this from automation altogether, and make releaseduty schedule it by hand

None are great options.

Comment 4

[[e-mail] (:bhearsum)]

Rail and I chatted this morning, and we're going to do Ship It integration for this. There's a bunch of pieces to touch, and we'll be splitting up that work:
* Ship It (rail)
* Release Runner (rail)
* Releasetasks (rail)
* The script that triggers graph2 (rail)
* Buildbot (bhearsum)
* publish_balrog.py (bhearsum)
* balrog-release-shipper.py (bhearsum)
* cli.py (bhearsum)
* balrogclient (bhearsum)

The majority of these changes are just to pass the scheduled time from ship it -> balrog. There's a couple of other notable parts:
* We decided that calling it "Expected Release Time" would be best, to avoid confusion in the case that we change the scheduled release time after the fact (Balrog would get the new one, but Ship It wouldn't).
* The current "publish to balrog" step will be renamed to something like "schedule release in balrog", and happen immediately after final verify (instead of after the human decision task. The human decision task will still need to run so that bouncer and other post-release activities happen.

Comment 5

[[e-mail] (:bhearsum)]

Attachment: add buildbot builder for scheduling publishing

Technically, this isn't needed because all of the factory details are the same as the existing "publish" builder, but it has a different name which might help avoid confusion. If you'd rather use the same builder that's cool with me.

Comment 6

[[e-mail] (:bhearsum)]

Attachment: pass along scheduled time and background rate in mozharness

This assumes we'll be receiving the scheduled time and background rate from buildbot proprerties. Because throttling is a per-channel configuration, we need to make it the responsibility of releasetasks to set the right value.

Comment 7

[[e-mail] (:bhearsum)]

Attachment: add scheduled rule change support to balrogclient

Comment 8

[[e-mail] (:bhearsum)]

Attachment: add support for scheduling publishing in the tools repo

There's a few parts to this patch:
* Import balrogclient changes to get API support
* Add "ReleaseScheduler" class to cli.py, which works much like ReleasePusher, but schedules a change with fallbackingMapping + backgroundRate set.
* Update balrog-release-shipper.py to support scheduling and direct modification of Rules.

I tested this against a local version of Balrog in schedule mode with background rate, schedule mode without background rate, direct update with background rate, and direct update without background rate.

Comment 9

[Simon Fraser [:sfraser] ⌚️GMT]

Comment on attachment 8851995 [details] [review]
add scheduled rule change support to balrogclient

Looks good to me, although I don't have write access so my review on github doesn't mean much.

Comment 10

[[e-mail] (:bhearsum)]

Another thing to double check: make sure ffxbld and tbirdbld can create scheduled changes.

Comment 11

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum (:bhearsum) from comment #10)
> Another thing to double check: make sure ffxbld and tbirdbld can create
> scheduled changes.

They can! No special permissions are needed to create scheduled changes (they inherit permissions from the object that they will be creating/updating/deleting).

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 8852001 [details] [diff] [review]
add support for scheduling publishing in the tools repo

Had to backout due to bustage:
07:25:19     INFO -  Traceback (most recent call last):
07:25:19     INFO -    File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/balrog-release-shipper.py", line 14, in <module>
07:25:19     INFO -      from balrog.submitter.cli import ReleasePusher, ReleaseScheduler
07:25:19     INFO -    File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/../../lib/python/balrog/submitter/cli.py", line 1, in <module>
07:25:19     INFO -      import arrow
07:25:19     INFO -    File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/../../lib/python/vendor/arrow-0.10.0/arrow/__init__.py", line 3, in <module>
07:25:19     INFO -      from .arrow import Arrow
07:25:19     INFO -    File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/../../lib/python/vendor/arrow-0.10.0/arrow/arrow.py", line 11, in <module>
07:25:19     INFO -      from dateutil import tz as dateutil_tz
07:25:19     INFO -  ImportError: No module named dateutil

Looks like I had a virtualenv active when I tested this.

Comment 13

[[e-mail] (:bhearsum)]

Attachment: tools repo patch w/ all required dependencies

Turns out I had python-dateutil and six installed in my local environment when I tested this, but our build machines don't. This patch includes arrow *and* its dependencies.

Comment 14

[[e-mail] (:bhearsum)]

Attachment: update mozharness patch

Just before landing this, I realized there was a bug - we would pass silly things like "--schedule-at None" to the tools script. This should fix that.

Comment 15

[[e-mail] (:bhearsum)]

Comment on attachment 8853481 [details] [diff] [review]
tools repo patch w/ all required dependencies

https://hg.mozilla.org/build/tools/rev/68fd4127c10464e0814ceb9994222da721a92321

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 8853492 [details] [diff] [review]
update mozharness patch

https://hg.mozilla.org/integration/mozilla-inbound/rev/aaf1a83844cf6375f6312d84933125899a6a9875
https://hg.mozilla.org/releases/mozilla-aurora/rev/e2aa37453df9b28e6b5e3302454900c571888f62
https://hg.mozilla.org/releases/mozilla-beta/rev/3a2e5c6e4ed33d9cb8b04b32f85b45a9aad4c8d8
https://hg.mozilla.org/releases/mozilla-release/rev/8b37e49ca5ca99353e6bf82e2ac83d6d4aaf5be8
https://hg.mozilla.org/releases/mozilla-esr45/rev/eb6f26e1954726e7167592714c96baf47dd3d7bb
https://hg.mozilla.org/releases/mozilla-esr52/rev/52ee31edc0b41eecf1a7bc8f46d1db75e78fbc61

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/cc02f8a2839a
Backout patch from bug 1347212 with that errantly referenced bug 1247212 instead.
https://hg.mozilla.org/integration/mozilla-inbound/rev/f0b4339da129
improve release automation publishing of releases - allow pushes to be scheduled instead of done directly. r=rail

Comment 18

[[e-mail] (:bhearsum)]

Had to push another buildbotcustom patch because of builder name length issues (I have no idea thy they didn't show up last week...): https://hg.mozilla.org/build/buildbotcustom/rev/164f55952ab0

Comment 19

[[e-mail] (:bhearsum)]

More bustage, this time because I didn't add imports to another script that uses cli.py: https://hg.mozilla.org/build/tools/rev/682ba1ff066a

Comment 20

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/f0b4339da129

Comment 21

[[e-mail] (:bhearsum)]

We have a small follow-up to fix e-mail notifications to not tell lies (https://bugzilla.mozilla.org/show_bug.cgi?id=1379170), but all of this worked fine for the latest Beta. Thanks to everyone who pitched in!