1482395 - Move the WNP rules in-tree and let Releng update them

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: L10N, enhancement)

Description

[Johan Lorenzo [:jlorenzo]]

(Follow up of bug 1473356 which was a first attempt to automate WNP from a Releng point of view) 


TL;DR: We may be able to reduce the amount of comments needed in Bugzilla to handle the WNP process. Current proposal: a config file living in https://hg.mozilla.org/releases/mozilla-release and riding the trains.



1. GENERAL OVERVIEW

Setting up the What's New Page (WNP) is a process that involves the coordination of 6 different teams:
* Marketing
* the team in change of Bedrock (www.mozilla.org)
* L10n
* Release Management
* Release Engineering
* QE

These teams must agree on:
* the WNP URL (we may have several URLs, just like we did in Firefox 57, where some people had the CEO's letter and some a smaller message)
* what locales will get to see this WNP
* what channels will get to see it. It's usually "release" and its testing equivalent "release-localtest" + "release-cdntest"
* what update paths will get to see the WNP. For instance, users who are on 61.0.1 and upgrading to 61.0.2 won't get the WNP; users on 60.0.1 will.



2. CURRENT PROCESS

The coordination happens asynchronously in Bugzilla (like bug 1463821). Data is passed via comments and attachments. Needinfos are used to get an action from others.



3. PROPOSAL


3.1. SCOPE 

It will be hard to get all interactions automated. The scope of this proposal is to move the source of truth to the trees (namely mozilla-central, mozilla-beta, and mozilla-release) and let it ride the trains. This only applies to the 4 piece of data mentioned above. The rest can remain processed in Bugzilla.


3.2. NEW WORKFLOW

The new workflow revolves around a config file stored in tree.

  a. Marketing initiate the process
  b. The Bedrock team implements the pages. Then, they update the config file with the expected URL on the production Bedrock. The config file is then pushed to mozilla-release (or whatever repo)
  c. The L10n team makes sure the new pages are localized. When a locale is complete, they add it to the config file. A locale can technically be added in between 2 release candidate as long as it's pushed to the tree.
  d. Marketing updates the config file to tell what users coming from which version get to see the WNP.
  e. Release Management kicks a new release off. At this stage, any update to the WNP requires either a manual action of Release Engineering or a new release.
  f. Release Engineering ensures automation (namely "taskgraph") process the config file and translate it into a Balrog release/rule.


3.3. TECHNICAL DESIGN

3.3.1. The config file

I think this config file can live under config/whats_new_page.yml. It can look like:

```
---
# %LOCALE% is automatically replaced by Balrog.
# {version} is processed by taskgraph before it reaches Balrog.
- url: "https://www.mozilla.org/%LOCALE%/firefox/{version}/whatsnew/"
  for:
    channels:
    - release
    - release-localtest
    - release-cdntest
    locales:
    - an
    - bg
    - bs
    - cak
    - cs
    - cy
    - da
    - de
    - dsb
    - en-US
    - eo
    - es-AR
    - es-CL
    - es-ES
    - es-MX
    - et
    - eu
    - fa
    - fr
    - fy-NL
    - gn
    - gu-IN
    - hi-IN
    - hsb
    - hu
    - hy-AM
    - ia
    - id
    - it
    - ja
    - ja-JP-mac
    - ka
    - ko
    - lij
    - ms
    - nb-NO
    - nl
    - nn-NO
    - pl
    - pt-BR
    - pt-PT
    - ro
    - ru
    - sk
    - sl
    - sq
    - sr
    - sv-SE
    - tr
    - uk
    - zh-CN
    - zh-TW
    versions: ["{version.major_number}"]  # e.g.: ["<61.0"]. {version.major_number} reflects the current version. This is done by taskgraph.
```

This file will be linted to ensure it's a correct YAML.


3.3.2. Taskgraph changes (Releng implementation)

The decision will read config/whats_new_page.yml and generate 2 new tasks:
a. At the end of the promote graph, a new task will copy the existing Balrog release (aka Balrog blob) to {blob_name}-No-WNP. This will be useful when we still want to update users to this version (like a watershed) but we don't want to show the outdated WNP anymore/
b. After the previous task ended, it will change {blob_name} to include the WNP data. The decision parses config/whats_new_page.yml and passes the data via the task payload. Then the worker running the task checks the payload is valid and modifies the blob.



4. PROS/CONS?


4.1. POTENTIAL DRAWBACKS: UPDATES AFTER THE RELEASE STARTED?

Release Engineering still has the ability to update the WNP rules right before the release ships. The update. The process is simpler than the current one[1] and just requires to edit exiting data (instead of crafting it from scratch). 


4.2. PRO: LESS DATA ON BUGZILLA

Thanks to this model, we can reduce the amount of important data on Bugzilla. Moreover, changes can be reviewed - which is not exactly the case, currently. Errors are less likely to happen because of a bad copy-and-paste.


4.2. PRO: LESS BACK-AND-FORTH

Teams can work more independently. For instance, RC1 can start even though the full list of locales hasn't been posted. Changes in this list don't require an action Release Engineering whenever RC2 starts. Another example: QE can start testing 




What do you all think?


[1] https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/release-promotion/desktop/howto-rc.md#wnp

Comment 1

[Johan Lorenzo [:jlorenzo]]

NI'ing one person per team. What do you guys think of this? Feel free to redirect!

Comment 2

[Francesco Lodolo [:flod]]

I see a couple of pros: visibility outside of releng/balrog, historical data is available.

But then comes one big cons, which makes me think this is not a good idea: this is not really data that rides the train, it's something that is available only a few weeks before a release.

Example: we set up a list of locales for 61. Where should this land? The only relevant tree is release, or beta if we land before merge.

Both trees need approval from Release Managers, and approval time could potentially cut into the time available for localization, unless we create a fast lane for approval and landing.

It's also unclear from the proposal who would be in charge of updating this file.

> * the WNP URL (we may have several URLs, just like we did in Firefox 57,
> where some people had the CEO's letter and some a smaller message)

The URL, as far as I remember, has never changed? I think redirects are managed on Bedrock's side, even for the 57 CEO letter.
https://www.mozilla.org/de/firefox/57.0/whatsnew/
https://www.mozilla.org/it/firefox/57.0/whatsnew/

(In reply to Johan Lorenzo [:jlorenzo] from comment #0)
> Teams can work more independently. For instance, RC1 can start even though
> the full list of locales hasn't been posted. Changes in this list don't
> require an action Release Engineering whenever RC2 starts. Another example:
> QE can start testing 

I'm not sure how/if this proposal changes the status quo. I don't think anything is blocking RC1, besides the fact that testing is not going to be meaningful.

Comment 3

[Craig Cook (:craigcook)]

Adding pmac, who is more familiar with the inner workings of bedrock and can offer a more expert opinion than I can.

Comment 4

[Paul [:pmac] McLanahan]

I'm unclear on the location of this YAML config file. Is it to be served on the web by bedrock itself (e.g. https://www.mozilla.org/configs/firefox/whatsnew.yml)? Or is the suggestion simply to have the bedrock team generate the file and push it somewhere? If it's the latter I don't really see how that gains much efficiency since we don't have any extra data really. The list of locales is determined by another team, the URL, as flod pointed out, is determined well in advance just with the version number (anything fancy happens with redirects in bedrock itself), and the channels and versions are decided by another team as well. I think it might be good to have such a file in the process, and have said file live in a public repo for everyone to be able to review and propose changes to, but I don't see much gain in having the Marketing Engineering team own the file. Though we'll be very happy to contribute to the file and be more involved in the process I think.

:erenaud please correct me if I'm wrong though.

Comment 5

[Johan Lorenzo [:jlorenzo]]

Thank you for your quick feedback! 

(In reply to Francesco Lodolo [:flod] from comment #2)> 
> Example: we set up a list of locales for 61. Where should this land? The
> only relevant tree is release, or beta if we land before merge.
You're right. The patch must occur on one of these 2 trees. I think the structure can ride the trains and the final list be landed on mozilla-beta or mozilla-release. I looked back at the 3 last releases (going further back is a difficult exercise because the data was implemented differently in Balrog), we had and I see at least 40 locales remained from a version to another. Here are the details:

+---------------+---------------------------------------------------------------+
|               |                    Previous major version                     |
| Major version +-------------------+---------------------+---------------------+
|               | Identical locales |    Added locales    |   Removed locales   |
+---------------+-------------------+---------------------+---------------------+
| Firefox 61    |         42        | 4 (lt, kab, vi, fi)  | 10 (cs, hi-IN, ro,  |
|               |                   |                     | eo, bg, bs, eu, ru, |
|               |                   |                     | an, hy-AM)          |
|               |                   |                     |                     |
| Firefox 60    |         43        | 9 (hi-IN, gn, lij,  | 1 (fi)               |
|               |                   | ko, sr, ro, id, ia, |                     |
|               |                   | eu)                 |                     |
+---------------+-------------------+---------------------+---------------------+


I think we can live a list of locales in mozilla-central and make the final call in mozilla-release.


> Both trees need approval from Release Managers, and approval time could
> potentially cut into the time available for localization, unless we create a
> fast lane for approval and landing.
I'm fine having a fast lane if Release Management agrees. Because this file doesn't end up in the product and is tested thoroughly by QE, I think we can set up this fast lane. I checked the past 3 releases, here's when the locale list was communicated:

* Firefox 61: Tuesday, June 19th (bug 1463821 comment 13). This was 1 day after the RC1 started.
* Firefox 60: Friday, April 27th (bug 1455010 comment 9) - the Friday before RC1 started. RC1 start on Mondays evening (UTC)
* Firefox 59: Friday, March 9th (bug 1444486 comment 1) - 4 days after RC1 started. It was the Friday before we shipped 59. We ship on Tuesdays afternoon (UTC)


> It's also unclear from the proposal who would be in charge of updating this
> file.
I'm sorry this wasn't clear. I tried to explain it in section "3.2. NEW WORKFLOW". I think:
* the L10n team should be in charge of updating the locale list,
* the Bedrock team in charge of updating the expected URL (if ever needed), 
* the Release Engineering team in charge of ensuring the content of this file gets reflected to Balrog.


> The URL, as far as I remember, has never changed? I think redirects are
> managed on Bedrock's side, even for the 57 CEO letter.
Oh that's right. You're right, the switch was done on the bedrock side, in the end. Thank you for correcting this mistake!



> (In reply to Johan Lorenzo [:jlorenzo] from comment #0)
> > Teams can work more independently. For instance, RC1 can start even though
> > the full list of locales hasn't been posted. Changes in this list don't
> > require an action Release Engineering whenever RC2 starts. Another example:
> > QE can start testing 
> 
> I'm not sure how/if this proposal changes the status quo. I don't think
> anything is blocking RC1, besides the fact that testing is not going to be
> meaningful.
First of all, I apologize, I didn't finish my last sentence. Let me write it down: Another example: QE can start testing the WNP without waiting on Release Engineering at each new build.

The status quo is changed this way: you shouldn't need Release Engineering to get the list of locales update. Today, every time Release Management kicks a new release off mozilla-release, a Release Engineer has to copy the list of locale from the bug and transform it into a JSON understandable by Balrog. This happens at RC1, RC2, RC3+, and at dot releases. If someone forgets (like I did last week in 61.0.2), QE gets blocked on the testing.
If this file lives in-tree and updated by whoever has jurisdiction on this piece of data, then you guys don't need Release Engineering anymore :)

Comment 6

[Johan Lorenzo [:jlorenzo]]

(In reply to Paul [:pmac] McLanahan ⏰ET needinfo? me from comment #4)
To me this file should live in https://hg.mozilla.org/releases/mozilla-release. If the WNP URL doesn't change from a release to another, then the Bedrock team doesn't have any action to do with this file. Per Flod's comment and yours, it seems this new process shouldn't change anything to you then. Please let me know if you have any other questions.

Comment 7

[Julien Cristau [:jcristau]]

Let me try to summarise to make sure I understand.

AIUI this proposal would change nothing for anyone but releng.  releng would maintain the wnp targetting rules (locales/channels/versions) in tree once that's locked down for a given release rather than having to copy and paste a blob of json in each release candidate or dot release blob in balrog.  The rules would still be modifiable in balrog after the initial setup if necessary.  But in the most common case this would mean less manual work on releng's part in the critical path for update testing.

Did I get that right?

Comment 8

[Johan Lorenzo [:jlorenzo]]

(In reply to Julien Cristau [:jcristau] from comment #7)
> Let me try to summarise to make sure I understand.
Sounds good. 


> AIUI this proposal would change nothing for anyone but releng.
Not exactly, for instance, the L10n team has to change the way they communicate locales. Instead of Bugzilla, they will update this config file. 


> releng would maintain the wnp targetting rules (locales/channels/versions) in tree 
Releng would maintain the pipeline that tranforms the config file to a Balrog release blob. Then, L10n would maintain the list of locales and Marketing would maintain the channels and versions - if needed.

> The rules would still be modifiable in balrog after the initial setup if
> necessary.  But in the most common case this would mean less manual work on
> releng's part in the critical path for update testing.
That's right.


Please let me know if that's clear enough.

Comment 9

[Cornel Ionce [:noni] [Hubs QA]]

From manual QE pov this seems to reduce the back-and-forth created during a WNP testing, thus being a plus.
It sounds great to have all the process documented and also to have historical data recorded.

In the current process an e-mail is sent to release-signoff letting us know that the build is available on a certain update channel so, as concerns: 
- How will the QE be aware when the WNP/updates are available certain channel? Will the e-mails still be sent?
- Will the full list of locales be posted/updated before we start our testing?

Comment 10

[Johan Lorenzo [:jlorenzo]]

Digging this conversation up. Let's first let Releng update mozilla-release whenever the WNP config is ready. This is a solution that was suggested by many. This will make the release process easier from a releng standpoint, without requiring any changes on the other teams' side. 

> - How will the QE be aware when the WNP/updates are available certain channel? Will the e-mails still be sent?
Good news! These emails will remain. They are sent no matter if the WNP rule is ready or not.

- Will the full list of locales be posted/updated before we start our testing?
Let's keep the list of locales posted on the bug for now, and see how the process goes.

Comment 11

[Tom Prince [:tomprince]]

Attachment: Bug 1482395: [mozrelease] Add support for generating update metadata; r?mtabara

Comment 12

[Tom Prince [:tomprince]]

Attachment: Bug 1482395: [mozrelease] Generate whats new blob and populate it during release; r?mtabara

Comment 13

[Tom Prince [:tomprince]]

Attachment: Bug 1482395: Allow python3 specific entries in virtualenv_packages.txt;

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b7f29f09e6ea
Allow python3 specific entries in virtualenv_packages.txt; r=nalexander

Comment 15

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/b7f29f09e6ea

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b09a44bf41cb
[mozrelease] Add support for generating update metadata; r=mtabara
https://hg.mozilla.org/integration/autoland/rev/3bf6a4a7e135
[mozrelease] Generate whats new blob and populate it during release; r=mtabara

Comment 17

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/b09a44bf41cb
https://hg.mozilla.org/mozilla-central/rev/3bf6a4a7e135

Comment 18

[Tom Prince [:tomprince]]

Attachment: Bug 1482395: [mozrelease] Fix product details links for beta and deveditions;

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dd2f3ac3a0ba
[mozrelease] Fix product details links for beta and deveditions; r=sfraser

Comment 20

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-beta/rev/563339c115b0
https://hg.mozilla.org/releases/mozilla-beta/rev/e0e9425868e0
https://hg.mozilla.org/releases/mozilla-beta/rev/88ab1bde6278
https://hg.mozilla.org/releases/mozilla-beta/rev/be8fbba20256

Comment 21

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/dd2f3ac3a0ba

Comment 22

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/18505b1a3afb
https://hg.mozilla.org/releases/mozilla-esr60/rev/e659d82e2f45
https://hg.mozilla.org/releases/mozilla-esr60/rev/8300b35286e0