Closed Bug 1347212 Opened 3 years ago Closed 3 years ago
improve release automation publishing of releases
1.54 KB, patch
|Details | Diff | Splinter Review|
42 bytes, text/x-github-pull-request
|Details | Review|
826.16 KB, patch
|Details | Diff | Splinter Review|
1.16 KB, patch
|Details | Diff | Splinter Review|
Currently, release automation is responsible for updating the live release rules (that is, the ones specified in "publish_rules" in files like https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/releases/updates_firefox_release.py#43) when we ship. There's a few issues with it: - Humans have to manually adjust the backgroundRate to 25% first (for release channel) - Humans need to remember to set fallbackMapping (for release channel) - Humans have to be around to it at all! - The current automation will break when Required Signoffs are added We can fix all of these things by updating the existing automation to: - Schedule a change when we push to cdntest, instead of directly manipulating the rule when we ship -- Rail suggested that we may be able to grab the desired ship time from Ship It, and set that as the "when" for the change - Make the automation set backgroundRate to 25%, (release channel only) - Make the automation set the current mapping to fallback mapping, so that users on n-2 or older versions can still update somewhere while updates are throttled (release channel only)
Rail, does ship it already contain the desired ship time?
No, but it shouldn't be too hard to add this.
Rail and I are going to talk more about Ship It integration next week. If we don't do that, other options include: * Try to guess in automation, but that seems kindof scary * Guess in automation and then add a manual step to set the publish time correctly by hand * Schedule the change right at publish time - but that would mean that relman would give the go, we would unblock the human decision task, and then they'd have to go signoff in balrog * Remove this from automation altogether, and make releaseduty schedule it by hand None are great options.
Rail and I chatted this morning, and we're going to do Ship It integration for this. There's a bunch of pieces to touch, and we'll be splitting up that work: * Ship It (rail) * Release Runner (rail) * Releasetasks (rail) * The script that triggers graph2 (rail) * Buildbot (bhearsum) * publish_balrog.py (bhearsum) * balrog-release-shipper.py (bhearsum) * cli.py (bhearsum) * balrogclient (bhearsum) The majority of these changes are just to pass the scheduled time from ship it -> balrog. There's a couple of other notable parts: * We decided that calling it "Expected Release Time" would be best, to avoid confusion in the case that we change the scheduled release time after the fact (Balrog would get the new one, but Ship It wouldn't). * The current "publish to balrog" step will be renamed to something like "schedule release in balrog", and happen immediately after final verify (instead of after the human decision task. The human decision task will still need to run so that bouncer and other post-release activities happen.
Technically, this isn't needed because all of the factory details are the same as the existing "publish" builder, but it has a different name which might help avoid confusion. If you'd rather use the same builder that's cool with me.
Assignee: nobody → bhearsum
Attachment #8851730 - Flags: review?(rail)
This assumes we'll be receiving the scheduled time and background rate from buildbot proprerties. Because throttling is a per-channel configuration, we need to make it the responsibility of releasetasks to set the right value.
Attachment #8851732 - Flags: review?(rail)
There's a few parts to this patch: * Import balrogclient changes to get API support * Add "ReleaseScheduler" class to cli.py, which works much like ReleasePusher, but schedules a change with fallbackingMapping + backgroundRate set. * Update balrog-release-shipper.py to support scheduling and direct modification of Rules. I tested this against a local version of Balrog in schedule mode with background rate, schedule mode without background rate, direct update with background rate, and direct update without background rate.
Attachment #8852001 - Flags: review?(rail)
Comment on attachment 8851995 [details] [review] add scheduled rule change support to balrogclient Looks good to me, although I don't have write access so my review on github doesn't mean much.
Attachment #8851995 - Flags: review?(sfraser) → review+
Another thing to double check: make sure ffxbld and tbirdbld can create scheduled changes.
(In reply to Ben Hearsum (:bhearsum) from comment #10) > Another thing to double check: make sure ffxbld and tbirdbld can create > scheduled changes. They can! No special permissions are needed to create scheduled changes (they inherit permissions from the object that they will be creating/updating/deleting).
Attachment #8851995 - Flags: checked-in+
Attachment #8851730 - Flags: checked-in+
Attachment #8852001 - Flags: checked-in+
Comment on attachment 8852001 [details] [diff] [review] add support for scheduling publishing in the tools repo Had to backout due to bustage: 07:25:19 INFO - Traceback (most recent call last): 07:25:19 INFO - File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/balrog-release-shipper.py", line 14, in <module> 07:25:19 INFO - from balrog.submitter.cli import ReleasePusher, ReleaseScheduler 07:25:19 INFO - File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/../../lib/python/balrog/submitter/cli.py", line 1, in <module> 07:25:19 INFO - import arrow 07:25:19 INFO - File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/../../lib/python/vendor/arrow-0.10.0/arrow/__init__.py", line 3, in <module> 07:25:19 INFO - from .arrow import Arrow 07:25:19 INFO - File "/builds/slave/rel-m-beta-fx_publish_balrog-0/build/tools/scripts/build-promotion/../../lib/python/vendor/arrow-0.10.0/arrow/arrow.py", line 11, in <module> 07:25:19 INFO - from dateutil import tz as dateutil_tz 07:25:19 INFO - ImportError: No module named dateutil Looks like I had a virtualenv active when I tested this.
Attachment #8852001 - Flags: checked-in+ → checked-in-
Turns out I had python-dateutil and six installed in my local environment when I tested this, but our build machines don't. This patch includes arrow *and* its dependencies.
Just before landing this, I realized there was a bug - we would pass silly things like "--schedule-at None" to the tools script. This should fix that.
Comment on attachment 8853481 [details] [diff] [review] tools repo patch w/ all required dependencies https://hg.mozilla.org/build/tools/rev/68fd4127c10464e0814ceb9994222da721a92321
Attachment #8853481 - Flags: checked-in+
Comment on attachment 8853492 [details] [diff] [review] update mozharness patch https://hg.mozilla.org/integration/mozilla-inbound/rev/aaf1a83844cf6375f6312d84933125899a6a9875 https://hg.mozilla.org/releases/mozilla-aurora/rev/e2aa37453df9b28e6b5e3302454900c571888f62 https://hg.mozilla.org/releases/mozilla-beta/rev/3a2e5c6e4ed33d9cb8b04b32f85b45a9aad4c8d8 https://hg.mozilla.org/releases/mozilla-release/rev/8b37e49ca5ca99353e6bf82e2ac83d6d4aaf5be8 https://hg.mozilla.org/releases/mozilla-esr45/rev/eb6f26e1954726e7167592714c96baf47dd3d7bb https://hg.mozilla.org/releases/mozilla-esr52/rev/52ee31edc0b41eecf1a7bc8f46d1db75e78fbc61
Attachment #8853492 - Flags: checked-in+
Pushed by email@example.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/cc02f8a2839a Backout patch from bug 1347212 with that errantly referenced bug 1247212 instead. https://hg.mozilla.org/integration/mozilla-inbound/rev/f0b4339da129 improve release automation publishing of releases - allow pushes to be scheduled instead of done directly. r=rail
Had to push another buildbotcustom patch because of builder name length issues (I have no idea thy they didn't show up last week...): https://hg.mozilla.org/build/buildbotcustom/rev/164f55952ab0
More bustage, this time because I didn't add imports to another script that uses cli.py: https://hg.mozilla.org/build/tools/rev/682ba1ff066a
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
We have a small follow-up to fix e-mail notifications to not tell lies (https://bugzilla.mozilla.org/show_bug.cgi?id=1379170), but all of this worked fine for the latest Beta. Thanks to everyone who pitched in!
Status: REOPENED → RESOLVED
Closed: 3 years ago → 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.