Closed Bug 1431799 Opened 6 years ago Closed 6 years ago

add balrog publishing task for beta channel for RCs

Categories

(Release Engineering :: Release Automation: Other, enhancement)

Product:
Release Engineering
Component:
Release Automation: Other
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mozilla, Assigned: mozilla)

References

Details

Attachments

(6 files, 2 obsolete files)

bug 1431799 - add `version` input to release promotion action.
59 bytes, text/x-review-board-request
bhearsum
: review+
Details
bug 1431799 - add RC secondary tasks.
59 bytes, text/x-review-board-request
bhearsum
: review+
Details
bug 1431799 - rename desktop_release_type -> release_type.
59 bytes, text/x-review-board-request
bhearsum
: review+
Details
bug 1431799 - add helper version functions in rr3.
59 bytes, text/x-review-board-request
mtabara
: review+
Details
bug 1431799 - rr3: add `version`; use rc relpro flavor; stop using desktop_release_type
59 bytes, text/x-review-board-request
mtabara
: review+
Details
bug 1431799 - add ship_fennec_rc relpro flavor.
59 bytes, text/x-review-board-request
bhearsum
: review+
Details 
https://trello.com/c/EdVjX6fA/274-add-balrog-publishing-task-for-beta-channel-for-rcs

From mtabara:

Good question, let me dig a bit.

Looking into last graph of 58.0 RC, https://tools.taskcluster.net/groups/ELmfX3koT0GN-In8__rgXA/tasks/GrZLLsuJSgOlhUBetsnwPw/details is the publishing to beta channel task.

That's https://github.com/mozilla-releng/releasetasks/blob/master/releasetasks/templates/desktop/publish_balrog.yml.tmpl from releasetasks which is BBB.

The BBB script that runs is configured in bbbcustom here https://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py#l2065 and runs under the hood this MH script https://hg.mozilla.org/releases/mozilla-release/file/tip/testing/mozharness/scripts/release/publish_balrog.py

The configs used from bb configs are https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/config.py#l2051 which are in fact https://hg.mozilla.org/releases/mozilla-release/file/tip/testing/mozharness/configs/releases/updates_firefox_release.py

It sounds like based on the configs, specifically, update_channels variable, we can have one or two publish_to_balrog tasks. So the key here is how we set this variable depending on the graphs/releases.

In releasetasks logic we have two graphs. First one where RC is published to beta and the second one that is published to release channel.

For the first week of RC, we're configuring the script here https://github.com/mozilla-releng/releasetasks/blob/master/releasetasks/templates/desktop/publish_balrog.yml.tmpl#L34 by feeding them via release-runner here https://hg.mozilla.org/build/tools/file/tip/buildfarm/release/release-runner.py#l521 . Those values are coming from https://hg.mozilla.org/build/tools/file/tip/buildfarm/release/release-runner.py#l407 where they are pre-processed based on https://hg.mozilla.org/build/tools/file/tip/buildfarm/release/release-runner.py#l138 + https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/config.py#l2118.

For the second week of RC, we're manually triggering that graph. The same release runner script is used but this times it takes them from https://hg.mozilla.org/build/tools/file/tip/buildfarm/release/release-runner.py#l420, hence from https://github.com/mozilla-releng/releasetasks/blob/master/releasetasks/release_configs/prod_mozilla-release_firefox_rc_graph_2.yml#L36
https://github.com/escapewindow/gecko-dev/commits/ship-rc
Started some cleanup, as well as adding new release promotion flavors. I may stick with this approach or shift depending on what I find.
Blocks: 1433536
Ben, what do you mean by https://hg.mozilla.org/build/tools/rev/9f2e155cf4dd#l1.37 ?

I'm planning on keying a bunch of behavior on the rc flag: whether to run beetmover, whether to run version bumping, etc. If we lump that in with all release types that include a beta partial, we'll probably run into issues.

We may need a 2nd flag: the rc flag for the above behavior, and enable-some-rc-behavior (name?) that we can turn on for non-rc's with beta partials.
Flags: needinfo?(bhearsum)
Attached patch start refactoring rr3 (obsolete) — — Splinter Review 
Sorry, I'm touching https://trello.com/c/vOP7fml4/282-1433487-update-releaserunner3-to-automatically-run-the-push-flavor-rather-than-promote-for-certain-release-types which you've signed up for. I need to also hack rr3 to switch the release_promotion_flavor for RC builds, and ended up addressing that first.

I split the rr3 patches into two because the 2nd half can't land until my in-tree changes land on central+beta (+release?).
Attachment #8945913 - Flags: feedback?(mtabara)
Lots of todo's in here.
Hm, patching ship-it-dev will also affect maple. Maybe I'll test via treeherder custom action until I'm closer to a working solution.
(In reply to Aki Sasaki [:aki] from comment #2)
> Ben, what do you mean by
> https://hg.mozilla.org/build/tools/rev/9f2e155cf4dd#l1.37 ?
> 
> I'm planning on keying a bunch of behavior on the rc flag: whether to run
> beetmover, whether to run version bumping, etc. If we lump that in with all
> release types that include a beta partial, we'll probably run into issues.
> 
> We may need a 2nd flag: the rc flag for the above behavior, and
> enable-some-rc-behavior (name?) that we can turn on for non-rc's with beta
> partials.

Basically what I'm saying is that we need to enable secondary update verify, final verify, and balrog publishing (when it exists) when we are shipping to beta. My understanding is that RCs always ship to Beta, and occaisonaly point releases do (when we ship them before the next b1 ships).

However, I just dug into this more and it looks like it's been ages since we shipped a point release to beta (AFAICT, it was 38.0.5, which might've been a super special case). It's probably OK to assume that point releases never ship to beta given this.
Flags: needinfo?(bhearsum)
Comment on attachment 8945913 [details] [diff] [review]
start refactoring rr3

Review of attachment 8945913 [details] [diff] [review]:
-----------------------------------------------------------------

Lgtm!

::: buildfarm/release/release-runner3.py
@@ +104,5 @@
> +            return True
> +        # RC release types will enable beta-channel testing &
> +        # shipping. We need this for all "final" releases
> +        # and also any releases that include a beta as a partial.
> +        # The assumption than "shipping to beta channel" always

tiny typo s,than,that?
Attachment #8945913 - Flags: feedback?(mtabara) → feedback+
I've been trying a number of RC graphs [1]... hitting a number of UV and secondary UV issues. I'm not entirely sure if those are expected, or if I'm choosing the wrong partial versions, or if I've broken something.

Currently hoping the latest RC promote graph [2] does better.

[1] https://docs.google.com/document/d/15L91N7yIPlIwALd-04_Ch-lw0AKgcQN6vPwdSGmEHvw/edit
[2] https://tools.taskcluster.net/groups/QnrkxtpBSDS6vSdjayfzLA
Keywords: leave-open
Comment on attachment 8947334 [details]
bug 1431799 - add `version` input to release promotion action.

https://reviewboard.mozilla.org/r/217066/#review222956

::: taskcluster/docs/parameters.rst:148
(Diff revision 1)
>  
>  ``build_number``
>     Specify the release promotion build number.
>  
> +``version``
> +   Specify the display version for release tasks. For releases, this is often a less specific version number than ``app_version``.

I think this is backwards - version is typically _more_ specific. Eg: 58.0b10, 52.6.0esr are both "versions" whose app versions are 58.0 and 52.6.0 respectively.

(See https://hg.mozilla.org/releases/mozilla-beta/file/tip/browser/config/version.txt vs https://hg.mozilla.org/releases/mozilla-beta/file/tip/browser/config/version_display.txt)

I'm not sure if it matters here, but in our automation we've historically distinguished between "version" and "display version". The former being a string the above, the latter being something more presentable (like "58.0 Beta 10").

::: taskcluster/taskgraph/actions/release_promotion.py:229
(Diff revision 1)
>  def release_promotion_action(parameters, input, task_group_id, task_id, task):
>      release_promotion_flavor = input['release_promotion_flavor']
> +    promotion_config = RELEASE_PROMOTION_CONFIG[release_promotion_flavor]
>      release_history = {}
> -    desktop_release_type = None
> +    product = promotion_config['product']
> +    desktop_release_type = promotion_config.get('desktop_release_type', '')

Minor nit: having this defined here makes it seem like it will be used in the next block of logic. It looks like the only use now is setting it in parameters - maybe it's better to assign it to it directly, further down?
Attachment #8947334 - Flags: review?(bhearsum) → review+
Comment on attachment 8947335 [details]
bug 1431799 - add RC secondary tasks.

https://reviewboard.mozilla.org/r/217068/#review222968

I think this looks OK. Unfortunately, your latest staging run failed at the updates builder, so none of the update or final verify builders ran.

::: taskcluster/taskgraph/target_tasks.py:373
(Diff revision 1)
>  @_target_task('ship_firefox')
>  def target_tasks_ship_firefox(full_task_graph, parameters, graph_config):
>      """Select the set of tasks required to ship firefox.
>      Previous build deps will be optimized out via action task."""
> +    is_rc = (parameters.get('desktop_release_type') == 'rc')
> +    if is_rc:

The idea here is that we need to pull all of the tasks from the "upstream" phase, which is different depending on whether we're shipping to beta vs release?
Attachment #8947335 - Flags: review?(bhearsum) → review+
(In reply to Ben Hearsum (:bhearsum) from comment #11)
> Comment on attachment 8947334 [details]
> bug 1431799 - add `version` input to release promotion action.
> 
> https://reviewboard.mozilla.org/r/217066/#review222956
> 
> ::: taskcluster/docs/parameters.rst:148
> (Diff revision 1)
> >  
> >  ``build_number``
> >     Specify the release promotion build number.
> >  
> > +``version``
> > +   Specify the display version for release tasks. For releases, this is often a less specific version number than ``app_version``.
> 
> I think this is backwards - version is typically _more_ specific. Eg:
> 58.0b10, 52.6.0esr are both "versions" whose app versions are 58.0 and
> 52.6.0 respectively.
> 
> (See
> https://hg.mozilla.org/releases/mozilla-beta/file/tip/browser/config/version.
> txt vs
> https://hg.mozilla.org/releases/mozilla-beta/file/tip/browser/config/
> version_display.txt)
> 
> I'm not sure if it matters here, but in our automation we've historically
> distinguished between "version" and "display version". The former being a
> string the above, the latter being something more presentable (like "58.0
> Beta 10").

Thanks! Fixed.

> 
> ::: taskcluster/taskgraph/actions/release_promotion.py:229
> (Diff revision 1)
> >  def release_promotion_action(parameters, input, task_group_id, task_id, task):
> >      release_promotion_flavor = input['release_promotion_flavor']
> > +    promotion_config = RELEASE_PROMOTION_CONFIG[release_promotion_flavor]
> >      release_history = {}
> > -    desktop_release_type = None
> > +    product = promotion_config['product']
> > +    desktop_release_type = promotion_config.get('desktop_release_type', '')
> 
> Minor nit: having this defined here makes it seem like it will be used in
> the next block of logic. It looks like the only use now is setting it in
> parameters - maybe it's better to assign it to it directly, further down?

Makes sense. I'm also going to rename desktop_release_type to release_type I think, since we'll use `rc` in Fennec.

(In reply to Ben Hearsum (:bhearsum) from comment #12)
> Comment on attachment 8947335 [details]
> bug 1431799 - add RC secondary tasks.
> 
> https://reviewboard.mozilla.org/r/217068/#review222968
> 
> I think this looks OK. Unfortunately, your latest staging run failed at the
> updates builder, so none of the update or final verify builders ran.

Trying https://tools.taskcluster.net/groups/UWD3qPjWS5qN3D4HQSGwIQ . If that fails I may need help with RC updates.

> ::: taskcluster/taskgraph/target_tasks.py:373
> (Diff revision 1)
> >  @_target_task('ship_firefox')
> >  def target_tasks_ship_firefox(full_task_graph, parameters, graph_config):
> >      """Select the set of tasks required to ship firefox.
> >      Previous build deps will be optimized out via action task."""
> > +    is_rc = (parameters.get('desktop_release_type') == 'rc')
> > +    if is_rc:
> 
> The idea here is that we need to pull all of the tasks from the "upstream"
> phase, which is different depending on whether we're shipping to beta vs
> release?

Correct. Adding some comments for clarity.
Comment on attachment 8947545 [details]
bug 1431799 - rename desktop_release_type -> release_type.

https://reviewboard.mozilla.org/r/217244/#review223088
Attachment #8947545 - Flags: review?(bhearsum) → review+
Attachment #8945913 - Attachment is obsolete: true
Attachment #8945934 - Attachment is obsolete: true
Comment on attachment 8947695 [details]
bug 1431799 - add helper version functions in rr3.

https://reviewboard.mozilla.org/r/217408/#review223652
Attachment #8947695 - Flags: review?(mtabara) → review+
Comment on attachment 8947696 [details]
bug 1431799 - rr3: add `version`; use rc relpro flavor; stop using desktop_release_type

https://reviewboard.mozilla.org/r/217410/#review223656
Attachment #8947696 - Flags: review?(mtabara) → review+
Comment on attachment 8948097 [details]
bug 1431799 - add ship_fennec_rc relpro flavor.

https://reviewboard.mozilla.org/r/217716/#review223748

::: taskcluster/ci/push-apk/kind.yml:40
(Diff revision 1)
>              by-project:
>                 mozilla-central: 'beta'
>                 mozilla-beta: 'rollout'
>                 mozilla-release: 'rollout'
>                 default: 'invalid'
> +         rc-google-play-track:

This (and the overriding of the non rc- version in the transform) looks like it exists to handle the fact that there's two different types of push-apks tasks we want to run, which have different values for google-play-track and rollout-percentage. If that's the case, isn't that normally something we handle be creating a separate job, and then filtering one or the other away?
(In reply to Ben Hearsum (:bhearsum) from comment #28)
> Comment on attachment 8948097 [details]
> bug 1431799 - add ship_fennec_rc relpro flavor.
> 
> https://reviewboard.mozilla.org/r/217716/#review223748
> 
> ::: taskcluster/ci/push-apk/kind.yml:40
> (Diff revision 1)
> >              by-project:
> >                 mozilla-central: 'beta'
> >                 mozilla-beta: 'rollout'
> >                 mozilla-release: 'rollout'
> >                 default: 'invalid'
> > +         rc-google-play-track:
> 
> This (and the overriding of the non rc- version in the transform) looks like
> it exists to handle the fact that there's two different types of push-apks
> tasks we want to run, which have different values for google-play-track and
> rollout-percentage. If that's the case, isn't that normally something we
> handle be creating a separate job, and then filtering one or the other away?

Reviewboard isn't publishing my comment:

That is the normal behavior, but we don't want the 2nd pushapk to run if we push the first time. Rather than going through hoops to try to disable the 2nd, non-rc-ship pushapk, we can optimize it out if it's the same kind and label. That's the path this patch takes.
(In reply to Aki Sasaki [:aki] from comment #29)
> (In reply to Ben Hearsum (:bhearsum) from comment #28)
> > Comment on attachment 8948097 [details]
> > bug 1431799 - add ship_fennec_rc relpro flavor.
> > 
> > https://reviewboard.mozilla.org/r/217716/#review223748
> > 
> > ::: taskcluster/ci/push-apk/kind.yml:40
> > (Diff revision 1)
> > >              by-project:
> > >                 mozilla-central: 'beta'
> > >                 mozilla-beta: 'rollout'
> > >                 mozilla-release: 'rollout'
> > >                 default: 'invalid'
> > > +         rc-google-play-track:
> > 
> > This (and the overriding of the non rc- version in the transform) looks like
> > it exists to handle the fact that there's two different types of push-apks
> > tasks we want to run, which have different values for google-play-track and
> > rollout-percentage. If that's the case, isn't that normally something we
> > handle be creating a separate job, and then filtering one or the other away?
> 
> Reviewboard isn't publishing my comment:
> 
> That is the normal behavior, but we don't want the 2nd pushapk to run if we
> push the first time. Rather than going through hoops to try to disable the
> 2nd, non-rc-ship pushapk, we can optimize it out if it's the same kind and
> label. That's the path this patch takes.

So we only ever want one push-apk task as part of any given release? If so, that feels much nicer.
That's correct, aiui.
If we want to upload a new version, we would re-promote and re-ship-rc. When we're sure the RC build is good for release, we can non-rc ship, which will version bump and mark as shipped, but it won't re-push to Google Play (the re-push would fail). So the current patch will automatically do this, by optimizing out the push-apk task from the ship_fennec flavor, in favor of the one in the ship_fennec_rc flavor.
Comment on attachment 8948097 [details]
bug 1431799 - add ship_fennec_rc relpro flavor.

https://reviewboard.mozilla.org/r/217716/#review223968
Attachment #8948097 - Flags: review?(bhearsum) → review+
Pushed by asasaki@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8e4a9d428474
add `version` input to release promotion action. r=bhearsum
https://hg.mozilla.org/integration/autoland/rev/20643471875c
add RC secondary tasks. r=bhearsum
https://hg.mozilla.org/integration/autoland/rev/0d7b85a8b190
rename desktop_release_type -> release_type. r=bhearsum
https://hg.mozilla.org/integration/autoland/rev/0e9ca8b75a82
add ship_fennec_rc relpro flavor. r=bhearsum
https://hg.mozilla.org/build/tools/rev/99b39094a9368707e8f6c0cb8965d49a957136f5
https://hg.mozilla.org/build/tools/rev/d867e7cc12e416b8fdcd7f00cf638b604d3937ce

I removed the automatic push_firefox because beetmover switches behavior based on phase.
We should split beetmover into explicit promote-phase and push-phase tasks.
Keywords: leave-open
Updated + restarted rr3 on bm83 + bm85.
Updated the wiki. Templates are in https://github.com/mozilla-releng/releasewarrior-2.0/pull/60 .
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: