1339706 - use l10n-changesets.json for all mobile l10n fx53+

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Aki Sasaki (not active)]

We have 3 types of locale files for mobile:

- all-locales is a flatfile with the locales to single-locale repack in CI/nightly
- maemo-locales is a flatfile with the locales to multilocale build in nightly
- the l10n changesets json with the locales, revision, and platform list (multi / single)

mozharness doesn't particularly care which one you use; they're all equally parseable.

Since we're going to do TC nightlies and TC releases in a very similar manner, and since we're bumping the l10n-changesets.json on mozilla-beta (meaning we can point at the in-tree copy during releases), let's point at that file for everything mobile on Fx53+.  Single locale, multilocale, and releases.

Comment 1

[Aki Sasaki (not active)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f0d69309bf4d

Comment 2

[Rail Aliiev [:rail]]

You also need to touch the merge day scripts, where we revert the list of locales when uplift to aurora:

https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/merge_day/central_to_aurora.py#59-64
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/merge_day/gecko_migration.py#328-334

Comment 3

[Aki Sasaki (not active)]

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #2)
> You also need to touch the merge day scripts, where we revert the list of
> locales when uplift to aurora:
> 
> https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/
> merge_day/central_to_aurora.py#59-64
> https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/
> merge_day/gecko_migration.py#328-334

I think I covered that in https://hg.mozilla.org/try/rev/4fe13b1059847803d3e9759a304fb2123f3b7829 :)

Comment 5

[Aki Sasaki (not active)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=82d26a467080

Comment 6

[Aki Sasaki (not active)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c1f0d9098e2c

Comment 7

[Aki Sasaki (not active)]

Axel, would you be open to this approach? It does require some l10n.ini and compare-locales changes.

Comment 8

[Aki Sasaki (not active)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=949743b0788e

Comment 9

[Aki Sasaki (not active)]

Attachment: bug 1339706 - use l10n-changesets.json for everything mobile l10n.

Review commit: https://reviewboard.mozilla.org/r/112932/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/112932/

Comment 10

[Aki Sasaki (not active)]

Attachment: bug 1339706 - update mozharness configs and scripts for l10n-changesets.json.

Review commit: https://reviewboard.mozilla.org/r/112934/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/112934/

Comment 11

[Aki Sasaki (not active)]

Attachment: bug 1339706 - remove maemo-locales.

This leaves a pointer at the new l10n-changesets.json.  After bug 1337825 lands, we can remove mobile/android/locales/all-locales as well.

Review commit: https://reviewboard.mozilla.org/r/112936/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/112936/

Comment 12

[Aki Sasaki (not active)]

Removing the ni? since it's covered by the r?

Comment 13

[Aki Sasaki (not active)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9141b6808681

Comment 14

[Axel Hecht [:Pike]]

Sorry, but we decided a while ago that we'll go the other way around, and filed bug 1280730 to implement that.

Comment 15

[Axel Hecht [:Pike]]

Comment on attachment 8837938 [details]
bug 1339706 - remove maemo-locales.

https://reviewboard.mozilla.org/r/112936/#review114486

Comment 16

[Axel Hecht [:Pike]]

Comment on attachment 8837936 [details]
bug 1339706 - use l10n-changesets.json for everything mobile l10n.

https://reviewboard.mozilla.org/r/112932/#review114488

Comment 17

[Aki Sasaki (not active)]

Attachment: bug 1339706 - l10n-bumper should build platforms from in-tree files. a=release

We were downloading the json from the dashboard, but the l10n team wants to discontinue that.  Instead, we download the revisions from the dashboard for beta and jamun, and use 'default' revisions on central and aurora.  We build the platform lists from all-locales and maemo-locales.

I updated the mozilla-beta config to be live, but since we have the mozharness repo and revision pinned in puppet, we will be able to pin to a new revision and point at the jamun config file.

Review commit: https://reviewboard.mozilla.org/r/113220/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/113220/

Comment 18

[Aki Sasaki (not active)]

Attachment: bug 1339706 - update mozharness configs and scripts for l10n-changesets.json. a=release

This points the multilocale builds at the in-tree l10n-changesets.json.

Update the merge day configs to know about l10n-changesets.json.  If we keep the aurora_to_beta configs, we'll need l10n-changesets.json bumping to happen on beta before merge day.

Review commit: https://reviewboard.mozilla.org/r/113222/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/113222/

Comment 19

[Aki Sasaki (not active)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=24aa838faba4

Comment 20

[Aki Sasaki (not active)]

Oops, I forgot to add l10n-changesets.json to the list of files.  Updated patchset incoming.

Comment 21

[Aki Sasaki (not active)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0ee6ec215c8a

Comment 22

[Aki Sasaki (not active)]

Attachment: bug 1339706 - add l10n-changesets.json. a=release

Review commit: https://reviewboard.mozilla.org/r/113262/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/113262/

Comment 23

[Aki Sasaki (not active)]

Comment on attachment 8838273 [details]
bug 1339706 - update mozharness configs and scripts for l10n-changesets.json.  a=release

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/113222/diff/1-2/

Comment 24

[Aki Sasaki (not active)]

Ok. Latest try graph went green for multilocale.  Signing failed on Try, because we don't have automatic scope setting yet (I have an idea how to do this) and Chain of Trust refuses to do nightly signing on Try.  Ready for review!

Comment 25

[Axel Hecht [:Pike]]

Comment on attachment 8838273 [details]
bug 1339706 - update mozharness configs and scripts for l10n-changesets.json.  a=release

https://reviewboard.mozilla.org/r/113222/#review114820

::: testing/mozharness/scripts/merge_day/gecko_migration.py:366
(Diff revision 2)
> +            self.run_command(
> +                hg + ["revert", "-r", end_tag, f],
> +                cwd=dirs['abs_to_dir'],
> +                error_list=HgErrorList,
> +                halt_on_failure=True,
> +            )

I think we could also invoke the l10n bouncer here, the l10n dashboard should have all the right data.

"Here" being after the version bump, I guess?

No idea if that's readily approachable from the merge script, just putting this out there.

Reverting the l10n-changesets may be OK for the B1 build, but obviously we really want to the l10n bouncer to have done its work before we go to build for B1.

Comment 26

[Axel Hecht [:Pike]]

I didn't do a full review, but I glanced at the patches, and haven't found anything unexpected after our chat yesterday.

Just left one food-for-thought about aurora->beta merge above.

Comment 27

[Rail Aliiev [:rail]]

Comment on attachment 8838272 [details]
bug 1339706 - l10n-bumper should build platforms from in-tree files.  a=release

https://reviewboard.mozilla.org/r/113220/#review114912

Comment 28

[Rail Aliiev [:rail]]

Comment on attachment 8838307 [details]
bug 1339706 - add l10n-changesets.json.  a=release

https://reviewboard.mozilla.org/r/113262/#review114914

Comment 29

[Rail Aliiev [:rail]]

Comment on attachment 8838273 [details]
bug 1339706 - update mozharness configs and scripts for l10n-changesets.json.  a=release

https://reviewboard.mozilla.org/r/113222/#review114916

::: testing/mozharness/scripts/merge_day/gecko_migration.py:366
(Diff revision 2)
> +            self.run_command(
> +                hg + ["revert", "-r", end_tag, f],
> +                cwd=dirs['abs_to_dir'],
> +                error_list=HgErrorList,
> +                halt_on_failure=True,
> +            )

I'm a bit concerned about this part. We rarely want to keep the same list of beta-only locales for multiple cycles, and sometimes we add new locales in aurora. If we have new locales in aurora, they would be killed by this block, but re-added right away in an hour or so after the next l10n bumper run. In this case we don't probably want to start a release build using the reverted list of locales and the old changesets. I'd say kill this block.

Otherwise lookg great to me!

Comment 30

[Aki Sasaki (not active)]

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #29)
> Comment on attachment 8838273 [details]
> bug 1339706 - update mozharness configs and scripts for l10n-changesets.json.
> 
> https://reviewboard.mozilla.org/r/113222/#review114916
> 
> ::: testing/mozharness/scripts/merge_day/gecko_migration.py:366
> (Diff revision 2)
> > +            self.run_command(
> > +                hg + ["revert", "-r", end_tag, f],
> > +                cwd=dirs['abs_to_dir'],
> > +                error_list=HgErrorList,
> > +                halt_on_failure=True,
> > +            )
> 
> I'm a bit concerned about this part. We rarely want to keep the same list of
> beta-only locales for multiple cycles, and sometimes we add new locales in
> aurora. If we have new locales in aurora, they would be killed by this
> block, but re-added right away in an hour or so after the next l10n bumper
> run. In this case we don't probably want to start a release build using the
> reverted list of locales and the old changesets. I'd say kill this block.

Ok. I'll submit a new patch.  This will mean that after we run the merge script and before the bumper runs, we'll have the same locale list as aurora, but they'll all be pinned to "default", which is not what we want on beta.  So we'll need to run the bumper after we run the merge day script either way.

Comment 31

[Aki Sasaki (not active)]

(In reply to Axel Hecht [:Pike] from comment #25)
> ::: testing/mozharness/scripts/merge_day/gecko_migration.py:366
> (Diff revision 2)
> > +            self.run_command(
> > +                hg + ["revert", "-r", end_tag, f],
> > +                cwd=dirs['abs_to_dir'],
> > +                error_list=HgErrorList,
> > +                halt_on_failure=True,
> > +            )
> 
> I think we could also invoke the l10n bouncer here, the l10n dashboard
> should have all the right data.
> 
> "Here" being after the version bump, I guess?
> 
> No idea if that's readily approachable from the merge script, just putting
> this out there.
> 
> Reverting the l10n-changesets may be OK for the B1 build, but obviously we
> really want to the l10n bouncer to have done its work before we go to build
> for B1.

I think I agree with you if we s,bouncer,bumper, :)  I think the bumper needs to run after the merge day script.

Comment 32

[Aki Sasaki (not active)]

Comment on attachment 8838273 [details]
bug 1339706 - update mozharness configs and scripts for l10n-changesets.json.  a=release

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/113222/diff/2-3/

Comment 33

[Rail Aliiev [:rail]]

Comment on attachment 8838273 [details]
bug 1339706 - update mozharness configs and scripts for l10n-changesets.json.  a=release

https://reviewboard.mozilla.org/r/113222/#review114956

Comment 34

[Aki Sasaki (not active)]

Comment on attachment 8838272 [details]
bug 1339706 - l10n-bumper should build platforms from in-tree files.  a=release

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/113220/diff/1-2/

Comment 35

[Aki Sasaki (not active)]

Comment on attachment 8838307 [details]
bug 1339706 - add l10n-changesets.json.  a=release

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/113262/diff/1-2/

Comment 36

[Aki Sasaki (not active)]

Comment on attachment 8838273 [details]
bug 1339706 - update mozharness configs and scripts for l10n-changesets.json.  a=release

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/113222/diff/3-4/

Comment 37

[Aki Sasaki (not active)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7fe88797039fe30fffce5a54b8a8abeb859bd039
bug 1339706 - l10n-bumper should build platforms from in-tree files. r=rail a=release

https://hg.mozilla.org/integration/mozilla-inbound/rev/9571f5beb2aeecd59aab1daf9438f1d496e8db59
bug 1339706 - add l10n-changesets.json. r=rail a=release

https://hg.mozilla.org/integration/mozilla-inbound/rev/75c813edfe863c48fd20b87e9b4a6e5f70cbb2b1
bug 1339706 - update mozharness configs and scripts for l10n-changesets.json. r=rail a=release

Comment 38

[Aki Sasaki (not active)]

https://hg.mozilla.org/projects/jamun/rev/08e73179637619eec3df1e5fa03f932be4c4b2dd
bug 1339706 - l10n-bumper should build platforms from in-tree files. r=rail a=release

https://hg.mozilla.org/projects/jamun/rev/5676335379597329b8c4208d3087ed02742df5dd
bug 1339706 - update mozharness configs and scripts for l10n-changesets.json. r=rail a=release

Comment 39

[Aki Sasaki (not active)]

Attachment: puppet-jamun-bumper.diff

Let's start bumping jamun with the new jamun revision.

Comment 40

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8838672 [details] [diff] [review]
puppet-jamun-bumper.diff

Sure you don't want to grab the latest one 8a757bcfe789 containing the multi configs fix?

Comment 41

[Aki Sasaki (not active)]

Either way. I just need the l10n bumper changes.

Comment 42

[Aki Sasaki (not active)]

https://hg.mozilla.org/build/puppet/rev/a89f43e27e218191c757d21b6072b2e0bff6e96d
bug 1339706 - update jamun l10n-bumper mozharness_revision. r=mtabara

Comment 43

[Aki Sasaki (not active)]

Attachment: jamun-config.diff

Oops, let's also use the new jamun config, rather than bumping mozilla-beta.

Comment 44

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8838701 [details] [diff] [review]
jamun-config.diff

Review of attachment 8838701 [details] [diff] [review]:
-----------------------------------------------------------------

Sorry for having missed that at prior r? ! O_O

Comment 45

[Aki Sasaki (not active)]

https://hg.mozilla.org/build/puppet/rev/72edbcfeba573eb32bfbf7ae296865ec66e1075c
bug 1339706 - use the jamun config for jamun l10n-bumper. r=mtabara

Comment 46

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/7fe88797039f
https://hg.mozilla.org/mozilla-central/rev/9571f5beb2ae
https://hg.mozilla.org/mozilla-central/rev/75c813edfe86

Comment 47

[Aki Sasaki (not active)]

Attachment: bumper-central-aurora.diff

Enable l10n_bumper on central and aurora.  This is based on the latest mozilla-central mozharness, which has configs for all branches.

This also removes the "creates" in download_mozharness, so a revision/repo change reruns the script even if /builds/l10n-bumper/mozharness already exists.

Comment 48

[Aki Sasaki (not active)]

https://hg.mozilla.org/build/puppet/rev/372c584ecf4c9bcd831069bb936ab70753561f61
bug 1339706 - l10n bumping on m-c and m-a. r=rail

Comment 50

[Aki Sasaki (not active)]

Status:

* update merge duty + release duty docs
* schedule mozilla-beta l10n-bumper

Those should happen sometime next week.  Otherwise we're done here.

Comment 51

[Aki Sasaki (not active)]

Attachment: m-b-l10n-bumper

Two things:
1. is it ok to use a linux test buildmaster for this? all the non-try build masters already have something else going on.
2. do you think it's best to turn this on today/tomorrow, so we have beta l10n bumper before merge day, or turn it on next monday?

Comment 52

[Aki Sasaki (not active)]

https://hg.mozilla.org/build/puppet/rev/74dc78db06ff1d9ef9b6fd3387f212c4fd66e13b
bug 1339706 - enable l10n-bumper on mozilla-beta. r=rail

Comment 53

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.