Closed Bug 1347212 Opened 3 years ago Closed 3 years ago

improve release automation publishing of releases


(Release Engineering :: Release Automation: Other, enhancement, P2)



(firefox-esr45 fixed, firefox-esr52 fixed, firefox53 fixed, firefox55 fixed)

Tracking Status
firefox-esr45 --- fixed
firefox-esr52 --- fixed
firefox53 --- fixed
firefox55 --- fixed


(Reporter: bhearsum, Assigned: bhearsum)




(4 files, 2 obsolete files)

Currently, release automation is responsible for updating the live release rules (that is, the ones specified in "publish_rules" in files like when we ship. There's a few issues with it:
- Humans have to manually adjust the backgroundRate to 25% first (for release channel)
- Humans need to remember to set fallbackMapping (for release channel)
- Humans have to be around to it at all!
- The current automation will break when Required Signoffs are added

We can fix all of these things by updating the existing automation to:
- Schedule a change when we push to cdntest, instead of directly manipulating the rule when we ship
-- Rail suggested that we may be able to grab the desired ship time from Ship It, and set that as the "when" for the change
- Make the automation set backgroundRate to 25%, (release channel only)
- Make the automation set the current mapping to fallback mapping, so that users on n-2 or older versions can still update somewhere while updates are throttled (release channel only)
Priority: -- → P2
Rail, does ship it already contain the desired ship time?
Flags: needinfo?(rail)
No, but it shouldn't be too hard to add this.
Flags: needinfo?(rail)
Rail and I are going to talk more about Ship It integration next week. If we don't do that, other options include:
* Try to guess in automation, but that seems kindof scary
* Guess in automation and then add a manual step to set the publish time correctly by hand
* Schedule the change right at publish time - but that would mean that relman would give the go, we would unblock the human decision task, and then they'd have to go signoff in balrog
* Remove this from automation altogether, and make releaseduty schedule it by hand

None are great options.
Rail and I chatted this morning, and we're going to do Ship It integration for this. There's a bunch of pieces to touch, and we'll be splitting up that work:
* Ship It (rail)
* Release Runner (rail)
* Releasetasks (rail)
* The script that triggers graph2 (rail)
* Buildbot (bhearsum)
* (bhearsum)
* (bhearsum)
* (bhearsum)
* balrogclient (bhearsum)

The majority of these changes are just to pass the scheduled time from ship it -> balrog. There's a couple of other notable parts:
* We decided that calling it "Expected Release Time" would be best, to avoid confusion in the case that we change the scheduled release time after the fact (Balrog would get the new one, but Ship It wouldn't).
* The current "publish to balrog" step will be renamed to something like "schedule release in balrog", and happen immediately after final verify (instead of after the human decision task. The human decision task will still need to run so that bouncer and other post-release activities happen.
Technically, this isn't needed because all of the factory details are the same as the existing "publish" builder, but it has a different name which might help avoid confusion. If you'd rather use the same builder that's cool with me.
Assignee: nobody → bhearsum
Attachment #8851730 - Flags: review?(rail)
This assumes we'll be receiving the scheduled time and background rate from buildbot proprerties. Because throttling is a per-channel configuration, we need to make it the responsibility of releasetasks to set the right value.
Attachment #8851732 - Flags: review?(rail)
There's a few parts to this patch:
* Import balrogclient changes to get API support
* Add "ReleaseScheduler" class to, which works much like ReleasePusher, but schedules a change with fallbackingMapping + backgroundRate set.
* Update to support scheduling and direct modification of Rules.

I tested this against a local version of Balrog in schedule mode with background rate, schedule mode without background rate, direct update with background rate, and direct update without background rate.
Attachment #8852001 - Flags: review?(rail)
Comment on attachment 8851995 [details] [review]
add scheduled rule change support to balrogclient

Looks good to me, although I don't have write access so my review on github doesn't mean much.
Attachment #8851995 - Flags: review?(sfraser) → review+
Another thing to double check: make sure ffxbld and tbirdbld can create scheduled changes.
Attachment #8852001 - Flags: review?(rail) → review+
Attachment #8851732 - Flags: review?(rail) → review+
Attachment #8851730 - Flags: review?(rail) → review+
(In reply to Ben Hearsum (:bhearsum) from comment #10)
> Another thing to double check: make sure ffxbld and tbirdbld can create
> scheduled changes.

They can! No special permissions are needed to create scheduled changes (they inherit permissions from the object that they will be creating/updating/deleting).
Attachment #8851995 - Flags: checked-in+
Attachment #8851730 - Flags: checked-in+
Attachment #8852001 - Flags: checked-in+
Comment on attachment 8852001 [details] [diff] [review]
add support for scheduling publishing in the tools repo

Had to backout due to bustage:
07:25:19     INFO -  Traceback (most recent call last):
07:25:19     INFO -    File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/", line 14, in <module>
07:25:19     INFO -      from balrog.submitter.cli import ReleasePusher, ReleaseScheduler
07:25:19     INFO -    File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/../../lib/python/balrog/submitter/", line 1, in <module>
07:25:19     INFO -      import arrow
07:25:19     INFO -    File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/../../lib/python/vendor/arrow-0.10.0/arrow/", line 3, in <module>
07:25:19     INFO -      from .arrow import Arrow
07:25:19     INFO -    File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/../../lib/python/vendor/arrow-0.10.0/arrow/", line 11, in <module>
07:25:19     INFO -      from dateutil import tz as dateutil_tz
07:25:19     INFO -  ImportError: No module named dateutil

Looks like I had a virtualenv active when I tested this.
Attachment #8852001 - Flags: checked-in+ → checked-in-
Turns out I had python-dateutil and six installed in my local environment when I tested this, but our build machines don't. This patch includes arrow *and* its dependencies.
Attachment #8852001 - Attachment is obsolete: true
Attachment #8853481 - Flags: review?(rail)
Attachment #8853481 - Flags: review?(rail) → review+
Just before landing this, I realized there was a bug - we would pass silly things like "--schedule-at None" to the tools script. This should fix that.
Attachment #8851732 - Attachment is obsolete: true
Attachment #8853492 - Flags: review?(rail)
Attachment #8853492 - Flags: review?(rail) → review+
Pushed by
Backout patch from bug 1347212 with that errantly referenced bug 1247212 instead.
improve release automation publishing of releases - allow pushes to be scheduled instead of done directly. r=rail
Had to push another buildbotcustom patch because of builder name length issues (I have no idea thy they didn't show up last week...):
More bustage, this time because I didn't add imports to another script that uses
Closed: 3 years ago
Resolution: --- → FIXED
Resolution: FIXED → ---
Blocks: 1278974
Depends on: 1373647
Depends on: 1373680
We have a small follow-up to fix e-mail notifications to not tell lies (, but all of this worked fine for the latest Beta. Thanks to everyone who pitched in!
Closed: 3 years ago3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.