1431799 - add balrog publishing task for beta channel for RCs

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement)

Description

[Aki Sasaki (not active)]

https://trello.com/c/EdVjX6fA/274-add-balrog-publishing-task-for-beta-channel-for-rcs

From mtabara:

Good question, let me dig a bit.

Looking into last graph of 58.0 RC, https://tools.taskcluster.net/groups/ELmfX3koT0GN-In8__rgXA/tasks/GrZLLsuJSgOlhUBetsnwPw/details is the publishing to beta channel task.

That's https://github.com/mozilla-releng/releasetasks/blob/master/releasetasks/templates/desktop/publish_balrog.yml.tmpl from releasetasks which is BBB.

The BBB script that runs is configured in bbbcustom here https://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py#l2065 and runs under the hood this MH script https://hg.mozilla.org/releases/mozilla-release/file/tip/testing/mozharness/scripts/release/publish_balrog.py

The configs used from bb configs are https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/config.py#l2051 which are in fact https://hg.mozilla.org/releases/mozilla-release/file/tip/testing/mozharness/configs/releases/updates_firefox_release.py

It sounds like based on the configs, specifically, update_channels variable, we can have one or two publish_to_balrog tasks. So the key here is how we set this variable depending on the graphs/releases.

In releasetasks logic we have two graphs. First one where RC is published to beta and the second one that is published to release channel.

For the first week of RC, we're configuring the script here https://github.com/mozilla-releng/releasetasks/blob/master/releasetasks/templates/desktop/publish_balrog.yml.tmpl#L34 by feeding them via release-runner here https://hg.mozilla.org/build/tools/file/tip/buildfarm/release/release-runner.py#l521 . Those values are coming from https://hg.mozilla.org/build/tools/file/tip/buildfarm/release/release-runner.py#l407 where they are pre-processed based on https://hg.mozilla.org/build/tools/file/tip/buildfarm/release/release-runner.py#l138 + https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/config.py#l2118.

For the second week of RC, we're manually triggering that graph. The same release runner script is used but this times it takes them from https://hg.mozilla.org/build/tools/file/tip/buildfarm/release/release-runner.py#l420, hence from https://github.com/mozilla-releng/releasetasks/blob/master/releasetasks/release_configs/prod_mozilla-release_firefox_rc_graph_2.yml#L36

Comment 1

[Aki Sasaki (not active)]

https://github.com/escapewindow/gecko-dev/commits/ship-rc
Started some cleanup, as well as adding new release promotion flavors. I may stick with this approach or shift depending on what I find.

Comment 2

[Aki Sasaki (not active)]

Ben, what do you mean by https://hg.mozilla.org/build/tools/rev/9f2e155cf4dd#l1.37 ?

I'm planning on keying a bunch of behavior on the rc flag: whether to run beetmover, whether to run version bumping, etc. If we lump that in with all release types that include a beta partial, we'll probably run into issues.

We may need a 2nd flag: the rc flag for the above behavior, and enable-some-rc-behavior (name?) that we can turn on for non-rc's with beta partials.

Comment 3

[Aki Sasaki (not active)]

Attachment: start refactoring rr3

Sorry, I'm touching https://trello.com/c/vOP7fml4/282-1433487-update-releaserunner3-to-automatically-run-the-push-flavor-rather-than-promote-for-certain-release-types which you've signed up for. I need to also hack rr3 to switch the release_promotion_flavor for RC builds, and ended up addressing that first.

I split the rr3 patches into two because the 2nd half can't land until my in-tree changes land on central+beta (+release?).

Comment 4

[Aki Sasaki (not active)]

Attachment: wip: rr3: version; rc relpro flavor; stop using desktop_release_type

Lots of todo's in here.

Comment 5

[Aki Sasaki (not active)]

Hm, patching ship-it-dev will also affect maple. Maybe I'll test via treeherder custom action until I'm closer to a working solution.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] from comment #2)
> Ben, what do you mean by
> https://hg.mozilla.org/build/tools/rev/9f2e155cf4dd#l1.37 ?
> 
> I'm planning on keying a bunch of behavior on the rc flag: whether to run
> beetmover, whether to run version bumping, etc. If we lump that in with all
> release types that include a beta partial, we'll probably run into issues.
> 
> We may need a 2nd flag: the rc flag for the above behavior, and
> enable-some-rc-behavior (name?) that we can turn on for non-rc's with beta
> partials.

Basically what I'm saying is that we need to enable secondary update verify, final verify, and balrog publishing (when it exists) when we are shipping to beta. My understanding is that RCs always ship to Beta, and occaisonaly point releases do (when we ship them before the next b1 ships).

However, I just dug into this more and it looks like it's been ages since we shipped a point release to beta (AFAICT, it was 38.0.5, which might've been a super special case). It's probably OK to assume that point releases never ship to beta given this.

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8945913 [details] [diff] [review]
start refactoring rr3

Review of attachment 8945913 [details] [diff] [review]:
-----------------------------------------------------------------

Lgtm!

::: buildfarm/release/release-runner3.py
@@ +104,5 @@
> +            return True
> +        # RC release types will enable beta-channel testing &
> +        # shipping. We need this for all "final" releases
> +        # and also any releases that include a beta as a partial.
> +        # The assumption than "shipping to beta channel" always

tiny typo s,than,that?

Comment 8

[Aki Sasaki (not active)]

I've been trying a number of RC graphs [1]... hitting a number of UV and secondary UV issues. I'm not entirely sure if those are expected, or if I'm choosing the wrong partial versions, or if I've broken something.

Currently hoping the latest RC promote graph [2] does better.

[1] https://docs.google.com/document/d/15L91N7yIPlIwALd-04_Ch-lw0AKgcQN6vPwdSGmEHvw/edit
[2] https://tools.taskcluster.net/groups/QnrkxtpBSDS6vSdjayfzLA

Comment 9

[Aki Sasaki (not active)]

Attachment: bug 1431799 - add `version` input to release promotion action.

also clean up and move more config to the promotion config.

Review commit: https://reviewboard.mozilla.org/r/217066/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/217066/

Comment 10

[Aki Sasaki (not active)]

Attachment: bug 1431799 - add RC secondary tasks.

Review commit: https://reviewboard.mozilla.org/r/217068/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/217068/

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 8947334 [details]
bug 1431799 - add `version` input to release promotion action.

https://reviewboard.mozilla.org/r/217066/#review222956

::: taskcluster/docs/parameters.rst:148
(Diff revision 1)
>  
>  ``build_number``
>     Specify the release promotion build number.
>  
> +``version``
> +   Specify the display version for release tasks. For releases, this is often a less specific version number than ``app_version``.

I think this is backwards - version is typically _more_ specific. Eg: 58.0b10, 52.6.0esr are both "versions" whose app versions are 58.0 and 52.6.0 respectively.

(See https://hg.mozilla.org/releases/mozilla-beta/file/tip/browser/config/version.txt vs https://hg.mozilla.org/releases/mozilla-beta/file/tip/browser/config/version_display.txt)

I'm not sure if it matters here, but in our automation we've historically distinguished between "version" and "display version". The former being a string the above, the latter being something more presentable (like "58.0 Beta 10").

::: taskcluster/taskgraph/actions/release_promotion.py:229
(Diff revision 1)
>  def release_promotion_action(parameters, input, task_group_id, task_id, task):
>      release_promotion_flavor = input['release_promotion_flavor']
> +    promotion_config = RELEASE_PROMOTION_CONFIG[release_promotion_flavor]
>      release_history = {}
> -    desktop_release_type = None
> +    product = promotion_config['product']
> +    desktop_release_type = promotion_config.get('desktop_release_type', '')

Minor nit: having this defined here makes it seem like it will be used in the next block of logic. It looks like the only use now is setting it in parameters - maybe it's better to assign it to it directly, further down?

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 8947335 [details]
bug 1431799 - add RC secondary tasks.

https://reviewboard.mozilla.org/r/217068/#review222968

I think this looks OK. Unfortunately, your latest staging run failed at the updates builder, so none of the update or final verify builders ran.

::: taskcluster/taskgraph/target_tasks.py:373
(Diff revision 1)
>  @_target_task('ship_firefox')
>  def target_tasks_ship_firefox(full_task_graph, parameters, graph_config):
>      """Select the set of tasks required to ship firefox.
>      Previous build deps will be optimized out via action task."""
> +    is_rc = (parameters.get('desktop_release_type') == 'rc')
> +    if is_rc:

The idea here is that we need to pull all of the tasks from the "upstream" phase, which is different depending on whether we're shipping to beta vs release?

Comment 14

[Aki Sasaki (not active)]

(In reply to Ben Hearsum (:bhearsum) from comment #11)
> Comment on attachment 8947334 [details]
> bug 1431799 - add `version` input to release promotion action.
> 
> https://reviewboard.mozilla.org/r/217066/#review222956
> 
> ::: taskcluster/docs/parameters.rst:148
> (Diff revision 1)
> >  
> >  ``build_number``
> >     Specify the release promotion build number.
> >  
> > +``version``
> > +   Specify the display version for release tasks. For releases, this is often a less specific version number than ``app_version``.
> 
> I think this is backwards - version is typically _more_ specific. Eg:
> 58.0b10, 52.6.0esr are both "versions" whose app versions are 58.0 and
> 52.6.0 respectively.
> 
> (See
> https://hg.mozilla.org/releases/mozilla-beta/file/tip/browser/config/version.
> txt vs
> https://hg.mozilla.org/releases/mozilla-beta/file/tip/browser/config/
> version_display.txt)
> 
> I'm not sure if it matters here, but in our automation we've historically
> distinguished between "version" and "display version". The former being a
> string the above, the latter being something more presentable (like "58.0
> Beta 10").

Thanks! Fixed.

> 
> ::: taskcluster/taskgraph/actions/release_promotion.py:229
> (Diff revision 1)
> >  def release_promotion_action(parameters, input, task_group_id, task_id, task):
> >      release_promotion_flavor = input['release_promotion_flavor']
> > +    promotion_config = RELEASE_PROMOTION_CONFIG[release_promotion_flavor]
> >      release_history = {}
> > -    desktop_release_type = None
> > +    product = promotion_config['product']
> > +    desktop_release_type = promotion_config.get('desktop_release_type', '')
> 
> Minor nit: having this defined here makes it seem like it will be used in
> the next block of logic. It looks like the only use now is setting it in
> parameters - maybe it's better to assign it to it directly, further down?

Makes sense. I'm also going to rename desktop_release_type to release_type I think, since we'll use `rc` in Fennec.

(In reply to Ben Hearsum (:bhearsum) from comment #12)
> Comment on attachment 8947335 [details]
> bug 1431799 - add RC secondary tasks.
> 
> https://reviewboard.mozilla.org/r/217068/#review222968
> 
> I think this looks OK. Unfortunately, your latest staging run failed at the
> updates builder, so none of the update or final verify builders ran.

Trying https://tools.taskcluster.net/groups/UWD3qPjWS5qN3D4HQSGwIQ . If that fails I may need help with RC updates.

> ::: taskcluster/taskgraph/target_tasks.py:373
> (Diff revision 1)
> >  @_target_task('ship_firefox')
> >  def target_tasks_ship_firefox(full_task_graph, parameters, graph_config):
> >      """Select the set of tasks required to ship firefox.
> >      Previous build deps will be optimized out via action task."""
> > +    is_rc = (parameters.get('desktop_release_type') == 'rc')
> > +    if is_rc:
> 
> The idea here is that we need to pull all of the tasks from the "upstream"
> phase, which is different depending on whether we're shipping to beta vs
> release?

Correct. Adding some comments for clarity.

Comment 15

[Aki Sasaki (not active)]

Attachment: bug 1431799 - rename desktop_release_type -> release_type.

This used to only be relevant to Devedition and Firefox releases.
In bug 1433536 we're going to add RC Fennec releases. Let's rename
the parameter now, for less parameters churn.

Review commit: https://reviewboard.mozilla.org/r/217244/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/217244/

Comment 16

[Aki Sasaki (not active)]

Comment on attachment 8947334 [details]
bug 1431799 - add `version` input to release promotion action.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/217066/diff/1-2/

Comment 17

[Aki Sasaki (not active)]

Comment on attachment 8947335 [details]
bug 1431799 - add RC secondary tasks.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/217068/diff/1-2/

Comment 18

[[e-mail] (:bhearsum)]

Comment on attachment 8947545 [details]
bug 1431799 - rename desktop_release_type -> release_type.

https://reviewboard.mozilla.org/r/217244/#review223088

Comment 19

[Aki Sasaki (not active)]

Attachment: bug 1431799 - add helper version functions in rr3.

Review commit: https://reviewboard.mozilla.org/r/217408/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/217408/

Comment 20

[Aki Sasaki (not active)]

Attachment: bug 1431799 - rr3: add `version`; use rc relpro flavor; stop using desktop_release_type

Review commit: https://reviewboard.mozilla.org/r/217410/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/217410/

Comment 21

[Aki Sasaki (not active)]

Attachment: bug 1431799 - add ship_fennec_rc relpro flavor.

Also add rc-{google-play-track,rollout-percentage} for RC pushapk.

One nice side effect of using the same push-apk kind: we don't re-run push-apk during the ship_fennec relpro flavor if we've run the ship_fennec_rc flavor with the same build. (Google Play would reject the same buildid.)

This is really for bug 1433536, but MozReview is forcing me to include this patch with the others for reasons.

Review commit: https://reviewboard.mozilla.org/r/217716/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/217716/

Comment 22

[Aki Sasaki (not active)]

Comment on attachment 8947334 [details]
bug 1431799 - add `version` input to release promotion action.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/217066/diff/2-3/

Comment 23

[Aki Sasaki (not active)]

Comment on attachment 8947335 [details]
bug 1431799 - add RC secondary tasks.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/217068/diff/2-3/

Comment 24

[Aki Sasaki (not active)]

Comment on attachment 8947545 [details]
bug 1431799 - rename desktop_release_type -> release_type.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/217244/diff/1-2/

Comment 25

[Aki Sasaki (not active)]

Comment on attachment 8947696 [details]
bug 1431799 - rr3: add `version`; use rc relpro flavor; stop using desktop_release_type

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/217410/diff/1-2/

Comment 26

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8947695 [details]
bug 1431799 - add helper version functions in rr3.

https://reviewboard.mozilla.org/r/217408/#review223652

Comment 27

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8947696 [details]
bug 1431799 - rr3: add `version`; use rc relpro flavor; stop using desktop_release_type

https://reviewboard.mozilla.org/r/217410/#review223656

Comment 28

[[e-mail] (:bhearsum)]

Comment on attachment 8948097 [details]
bug 1431799 - add ship_fennec_rc relpro flavor.

https://reviewboard.mozilla.org/r/217716/#review223748

::: taskcluster/ci/push-apk/kind.yml:40
(Diff revision 1)
>              by-project:
>                 mozilla-central: 'beta'
>                 mozilla-beta: 'rollout'
>                 mozilla-release: 'rollout'
>                 default: 'invalid'
> +         rc-google-play-track:

This (and the overriding of the non rc- version in the transform) looks like it exists to handle the fact that there's two different types of push-apks tasks we want to run, which have different values for google-play-track and rollout-percentage. If that's the case, isn't that normally something we handle be creating a separate job, and then filtering one or the other away?

Comment 29

[Aki Sasaki (not active)]

(In reply to Ben Hearsum (:bhearsum) from comment #28)
> Comment on attachment 8948097 [details]
> bug 1431799 - add ship_fennec_rc relpro flavor.
> 
> https://reviewboard.mozilla.org/r/217716/#review223748
> 
> ::: taskcluster/ci/push-apk/kind.yml:40
> (Diff revision 1)
> >              by-project:
> >                 mozilla-central: 'beta'
> >                 mozilla-beta: 'rollout'
> >                 mozilla-release: 'rollout'
> >                 default: 'invalid'
> > +         rc-google-play-track:
> 
> This (and the overriding of the non rc- version in the transform) looks like
> it exists to handle the fact that there's two different types of push-apks
> tasks we want to run, which have different values for google-play-track and
> rollout-percentage. If that's the case, isn't that normally something we
> handle be creating a separate job, and then filtering one or the other away?

Reviewboard isn't publishing my comment:

That is the normal behavior, but we don't want the 2nd pushapk to run if we push the first time. Rather than going through hoops to try to disable the 2nd, non-rc-ship pushapk, we can optimize it out if it's the same kind and label. That's the path this patch takes.

Comment 30

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] from comment #29)
> (In reply to Ben Hearsum (:bhearsum) from comment #28)
> > Comment on attachment 8948097 [details]
> > bug 1431799 - add ship_fennec_rc relpro flavor.
> > 
> > https://reviewboard.mozilla.org/r/217716/#review223748
> > 
> > ::: taskcluster/ci/push-apk/kind.yml:40
> > (Diff revision 1)
> > >              by-project:
> > >                 mozilla-central: 'beta'
> > >                 mozilla-beta: 'rollout'
> > >                 mozilla-release: 'rollout'
> > >                 default: 'invalid'
> > > +         rc-google-play-track:
> > 
> > This (and the overriding of the non rc- version in the transform) looks like
> > it exists to handle the fact that there's two different types of push-apks
> > tasks we want to run, which have different values for google-play-track and
> > rollout-percentage. If that's the case, isn't that normally something we
> > handle be creating a separate job, and then filtering one or the other away?
> 
> Reviewboard isn't publishing my comment:
> 
> That is the normal behavior, but we don't want the 2nd pushapk to run if we
> push the first time. Rather than going through hoops to try to disable the
> 2nd, non-rc-ship pushapk, we can optimize it out if it's the same kind and
> label. That's the path this patch takes.

So we only ever want one push-apk task as part of any given release? If so, that feels much nicer.

Comment 31

[Aki Sasaki (not active)]

That's correct, aiui.
If we want to upload a new version, we would re-promote and re-ship-rc. When we're sure the RC build is good for release, we can non-rc ship, which will version bump and mark as shipped, but it won't re-push to Google Play (the re-push would fail). So the current patch will automatically do this, by optimizing out the push-apk task from the ship_fennec flavor, in favor of the one in the ship_fennec_rc flavor.

Comment 32

[[e-mail] (:bhearsum)]

Comment on attachment 8948097 [details]
bug 1431799 - add ship_fennec_rc relpro flavor.

https://reviewboard.mozilla.org/r/217716/#review223968

Comment 33

[Aki Sasaki (not active)]

Comment on attachment 8947695 [details]
bug 1431799 - add helper version functions in rr3.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/217408/diff/1-2/

Comment 34

[Aki Sasaki (not active)]

Comment on attachment 8947696 [details]
bug 1431799 - rr3: add `version`; use rc relpro flavor; stop using desktop_release_type

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/217410/diff/2-3/

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8e4a9d428474
add `version` input to release promotion action. r=bhearsum
https://hg.mozilla.org/integration/autoland/rev/20643471875c
add RC secondary tasks. r=bhearsum
https://hg.mozilla.org/integration/autoland/rev/0d7b85a8b190
rename desktop_release_type -> release_type. r=bhearsum
https://hg.mozilla.org/integration/autoland/rev/0e9ca8b75a82
add ship_fennec_rc relpro flavor. r=bhearsum

Comment 36

[Aki Sasaki (not active)]

https://hg.mozilla.org/build/tools/rev/99b39094a9368707e8f6c0cb8965d49a957136f5
https://hg.mozilla.org/build/tools/rev/d867e7cc12e416b8fdcd7f00cf638b604d3937ce

I removed the automatic push_firefox because beetmover switches behavior based on phase.
We should split beetmover into explicit promote-phase and push-phase tasks.

Comment 37

[Aki Sasaki (not active)]

https://hg.mozilla.org/releases/mozilla-beta/rev/23d8e662a20348d7607d2e6803cb1d30881c29cf
https://hg.mozilla.org/releases/mozilla-beta/rev/badd10b09c9a987061e116af4f64d7c9bdff5697
https://hg.mozilla.org/releases/mozilla-beta/rev/b6ba8f32b192bd25a912159fb5b10562921d7507
https://hg.mozilla.org/releases/mozilla-beta/rev/99f0936f7d8ce1bc7ae6d5d5c513fb047ee4214e

Comment 38

[Aki Sasaki (not active)]

Updated + restarted rr3 on bm83 + bm85.

Comment 39

[Aki Sasaki (not active)]

Updated the wiki. Templates are in https://github.com/mozilla-releng/releasewarrior-2.0/pull/60 .

Comment 40

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/8e4a9d428474
https://hg.mozilla.org/mozilla-central/rev/20643471875c
https://hg.mozilla.org/mozilla-central/rev/0d7b85a8b190
https://hg.mozilla.org/mozilla-central/rev/0e9ca8b75a82