improve release automation publishing of releases

RESOLVED FIXED

Status

enhancement
P2
normal
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: bhearsum, Assigned: bhearsum)

Tracking

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr45 fixed, firefox-esr52 fixed, firefox53 fixed, firefox55 fixed)

Details

Attachments

(4 attachments, 2 obsolete attachments)

Currently, release automation is responsible for updating the live release rules (that is, the ones specified in "publish_rules" in files like https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/releases/updates_firefox_release.py#43) when we ship. There's a few issues with it:
- Humans have to manually adjust the backgroundRate to 25% first (for release channel)
- Humans need to remember to set fallbackMapping (for release channel)
- Humans have to be around to it at all!
- The current automation will break when Required Signoffs are added

We can fix all of these things by updating the existing automation to:
- Schedule a change when we push to cdntest, instead of directly manipulating the rule when we ship
-- Rail suggested that we may be able to grab the desired ship time from Ship It, and set that as the "when" for the change
- Make the automation set backgroundRate to 25%, (release channel only)
- Make the automation set the current mapping to fallback mapping, so that users on n-2 or older versions can still update somewhere while updates are throttled (release channel only)

Updated

2 years ago
Priority: -- → P2
(Assignee)

Comment 1

2 years ago
Rail, does ship it already contain the desired ship time?
Flags: needinfo?(rail)
No, but it shouldn't be too hard to add this.
Flags: needinfo?(rail)
(Assignee)

Comment 3

2 years ago
Rail and I are going to talk more about Ship It integration next week. If we don't do that, other options include:
* Try to guess in automation, but that seems kindof scary
* Guess in automation and then add a manual step to set the publish time correctly by hand
* Schedule the change right at publish time - but that would mean that relman would give the go, we would unblock the human decision task, and then they'd have to go signoff in balrog
* Remove this from automation altogether, and make releaseduty schedule it by hand

None are great options.
(Assignee)

Comment 4

2 years ago
Rail and I chatted this morning, and we're going to do Ship It integration for this. There's a bunch of pieces to touch, and we'll be splitting up that work:
* Ship It (rail)
* Release Runner (rail)
* Releasetasks (rail)
* The script that triggers graph2 (rail)
* Buildbot (bhearsum)
* publish_balrog.py (bhearsum)
* balrog-release-shipper.py (bhearsum)
* cli.py (bhearsum)
* balrogclient (bhearsum)

The majority of these changes are just to pass the scheduled time from ship it -> balrog. There's a couple of other notable parts:
* We decided that calling it "Expected Release Time" would be best, to avoid confusion in the case that we change the scheduled release time after the fact (Balrog would get the new one, but Ship It wouldn't).
* The current "publish to balrog" step will be renamed to something like "schedule release in balrog", and happen immediately after final verify (instead of after the human decision task. The human decision task will still need to run so that bouncer and other post-release activities happen.
(Assignee)

Comment 5

2 years ago
Technically, this isn't needed because all of the factory details are the same as the existing "publish" builder, but it has a different name which might help avoid confusion. If you'd rather use the same builder that's cool with me.
Assignee: nobody → bhearsum
Attachment #8851730 - Flags: review?(rail)
(Assignee)

Comment 6

2 years ago
This assumes we'll be receiving the scheduled time and background rate from buildbot proprerties. Because throttling is a per-channel configuration, we need to make it the responsibility of releasetasks to set the right value.
Attachment #8851732 - Flags: review?(rail)
(Assignee)

Comment 8

2 years ago
There's a few parts to this patch:
* Import balrogclient changes to get API support
* Add "ReleaseScheduler" class to cli.py, which works much like ReleasePusher, but schedules a change with fallbackingMapping + backgroundRate set.
* Update balrog-release-shipper.py to support scheduling and direct modification of Rules.

I tested this against a local version of Balrog in schedule mode with background rate, schedule mode without background rate, direct update with background rate, and direct update without background rate.
Attachment #8852001 - Flags: review?(rail)
Comment on attachment 8851995 [details] [review]
add scheduled rule change support to balrogclient

Looks good to me, although I don't have write access so my review on github doesn't mean much.
Attachment #8851995 - Flags: review?(sfraser) → review+
(Assignee)

Comment 10

2 years ago
Another thing to double check: make sure ffxbld and tbirdbld can create scheduled changes.

Updated

2 years ago
Attachment #8852001 - Flags: review?(rail) → review+

Updated

2 years ago
Attachment #8851732 - Flags: review?(rail) → review+

Updated

2 years ago
Attachment #8851730 - Flags: review?(rail) → review+
(Assignee)

Comment 11

2 years ago
(In reply to Ben Hearsum (:bhearsum) from comment #10)
> Another thing to double check: make sure ffxbld and tbirdbld can create
> scheduled changes.

They can! No special permissions are needed to create scheduled changes (they inherit permissions from the object that they will be creating/updating/deleting).
(Assignee)

Updated

2 years ago
Attachment #8851995 - Flags: checked-in+
(Assignee)

Updated

2 years ago
Attachment #8851730 - Flags: checked-in+
(Assignee)

Updated

2 years ago
Attachment #8852001 - Flags: checked-in+
(Assignee)

Comment 12

2 years ago
Comment on attachment 8852001 [details] [diff] [review]
add support for scheduling publishing in the tools repo

Had to backout due to bustage:
07:25:19     INFO -  Traceback (most recent call last):
07:25:19     INFO -    File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/balrog-release-shipper.py", line 14, in <module>
07:25:19     INFO -      from balrog.submitter.cli import ReleasePusher, ReleaseScheduler
07:25:19     INFO -    File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/../../lib/python/balrog/submitter/cli.py", line 1, in <module>
07:25:19     INFO -      import arrow
07:25:19     INFO -    File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/../../lib/python/vendor/arrow-0.10.0/arrow/__init__.py", line 3, in <module>
07:25:19     INFO -      from .arrow import Arrow
07:25:19     INFO -    File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/../../lib/python/vendor/arrow-0.10.0/arrow/arrow.py", line 11, in <module>
07:25:19     INFO -      from dateutil import tz as dateutil_tz
07:25:19     INFO -  ImportError: No module named dateutil

Looks like I had a virtualenv active when I tested this.
Attachment #8852001 - Flags: checked-in+ → checked-in-
(Assignee)

Comment 13

2 years ago
Turns out I had python-dateutil and six installed in my local environment when I tested this, but our build machines don't. This patch includes arrow *and* its dependencies.
Attachment #8852001 - Attachment is obsolete: true
Attachment #8853481 - Flags: review?(rail)

Updated

2 years ago
Attachment #8853481 - Flags: review?(rail) → review+
(Assignee)

Comment 14

2 years ago
Just before landing this, I realized there was a bug - we would pass silly things like "--schedule-at None" to the tools script. This should fix that.
Attachment #8851732 - Attachment is obsolete: true
Attachment #8853492 - Flags: review?(rail)

Updated

2 years ago
Attachment #8853492 - Flags: review?(rail) → review+

Comment 17

2 years ago
Pushed by bhearsum@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/cc02f8a2839a
Backout patch from bug 1347212 with that errantly referenced bug 1247212 instead.
https://hg.mozilla.org/integration/mozilla-inbound/rev/f0b4339da129
improve release automation publishing of releases - allow pushes to be scheduled instead of done directly. r=rail
(Assignee)

Comment 18

2 years ago
Had to push another buildbotcustom patch because of builder name length issues (I have no idea thy they didn't show up last week...): https://hg.mozilla.org/build/buildbotcustom/rev/164f55952ab0
(Assignee)

Comment 19

2 years ago
More bustage, this time because I didn't add imports to another script that uses cli.py: https://hg.mozilla.org/build/tools/rev/682ba1ff066a

Comment 20

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/f0b4339da129
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Updated

2 years ago
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Updated

2 years ago
Blocks: 1278974

Updated

2 years ago
Depends on: 1373647
Depends on: 1373680
(Assignee)

Comment 21

2 years ago
We have a small follow-up to fix e-mail notifications to not tell lies (https://bugzilla.mozilla.org/show_bug.cgi?id=1379170), but all of this worked fine for the latest Beta. Thanks to everyone who pitched in!
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.