Closed Bug 812833 Opened 7 years ago Closed 7 years ago

Use multilocale gecko and gaia for b2g device builds (unagi, otoro, panda)

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(blocking-basecamp:+, b2g18 fixed)

RESOLVED FIXED
B2G C2 (20nov-10dec)
blocking-basecamp +
Tracking Status
b2g18 --- fixed

People

(Reporter: stas, Assigned: bhearsum)

References

Details

Attachments

(8 files, 5 obsolete files)

659 bytes, patch
aki
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
5.32 KB, patch
aki
: review+
bhearsum
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
2.49 KB, patch
aki
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
1.79 KB, patch
catlee
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
1.85 KB, patch
catlee
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
1.84 KB, patch
catlee
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
13.90 KB, patch
aki
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
12.19 KB, patch
catlee
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
When we currently create B2G builds, we simply use the locales that are present in git:  en-US, ar, fr, zh-TW.

I'd like to change this.

1. Let's build with en-US, ar, fr, zh-TW, es, pt-BR
2. For all five non en-US locales above, let's use the most up-to-date translations from http://hg.mozilla.org/gaia-l10n instead of the files which landed in git.

gaia locales: en-US, ar, fr, zh-TW, es, pt-BR
gecko locales: all Fennec locales
Assignee: nobody → bhearsum
The current unagi builds are Mozharness based. Here's a rough outline of what I think we need to do:
* Add an action to https://github.com/mozilla/build-mozharness/blob/master/scripts/b2g_build.py that will clone l10n repos. It will need to run after Gaia is cloned, and be pointed at a language.json file to read the list of locales it needs.
* Pass something to the build system to tell Gaia which language.json file to use. Might need to be passed indirectly through https://github.com/mozilla-b2g/B2G/blob/master/flash.sh#L174
* Pass something to the build system to indicate which directory the l10n repos will be in. Again, this may have to be forwarded in flash.sh

After those things, we should end up with a multilocale unagi build. Still not sure where we're going to alter the existing nightly unagi build for this or add a new one.
Note: We'll be modifying the existing RelEng unagi builds to include these these 6 locales. This was acked by Dietrich and Stas in e-mail.
Depends on the mozharness script changes posted in bug 812836. Languages filename may change.
Attachment #684478 - Flags: review?(aki)
Comment on attachment 684478 [details] [diff] [review]
pass --gaia-languages-file for unagi builds

Is the path correct?
i.e., do you need a build/ in front of there or anything?

If it helps to avoid thinking about cwd and relative paths, I'd be fine with either passing --gaia-languages-file with either a WithProperties or a %(abs_work_dir)s/ that mozharness replaces, or setting this in http://hg.mozilla.org/build/mozharness/file/2f1d18c735af/configs/b2g/releng-try.py and http://hg.mozilla.org/build/mozharness/file/2f1d18c735af/configs/b2g/releng.py if that makes sense.

I'm also fine with figuring out the correct relative path, and if this isn't it, correcting it.

So r=me if the correct path's getting to the script.
Attachment #684478 - Flags: review?(aki) → review+
(In reply to Aki Sasaki [:aki] from comment #4)
> Comment on attachment 684478 [details] [diff] [review]
> pass --gaia-languages-file for unagi builds
> 
> Is the path correct?
> i.e., do you need a build/ in front of there or anything?

--gaia-languages-file is a file that only exists after checking out Gaia. Because of that, this is a path relative to the Gaia checkout. I think this is the most precise we can get.
Depends on: 815189
blocking-basecamp: --- → +
Target Milestone: --- → B2G C2 (20nov-10dec)
Comment on attachment 684478 [details] [diff] [review]
pass --gaia-languages-file for unagi builds

This got rolled into the config updates in bug 812836.
Attachment #684478 - Attachment is obsolete: true
Now that bug 816977 is fixed I think it's safe to do this. Otoro is blocked on figuring why they're bricking.
Attachment #687867 - Flags: review?(aki)
Attachment #687867 - Flags: review?(aki) → review+
Attachment #687867 - Flags: checked-in+
Blocks: 817197
No longer blocks: 817197
Depends on: 817197
Depends on: 818032
Comment on attachment 687867 [details] [diff] [review]
re-enable panda and unagi multilocale

Had to back this out again because of this problem, that was first noticed in bug 812836, but never fully debugged. Filed bug 818032 on the issue.

Also, Rail told me that this patch is incomplete now because of overrides to the extra args on mozilla-beta: https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/b2g_config.py#L778
Attachment #687867 - Attachment is obsolete: true
Attachment #687867 - Flags: checked-in+ → checked-in-
Just want to summarize the state of this bug because it's C2 and not finished yet.

With bug 817108 being resolved as INVALID, we're waiting for bug 818032 to be fixed, then we can enabled these builds in production. Doing so is a simple flip of a switch.

Still have a single locale gecko pending bug 817197. This may end up not being a blocker, that's still in discussion.
No longer blocks: 818560
Depends on: 818560
Depends on: 817170
bug 817170 is changing the Gaia build system in such a way that we need to use an absolute path to the gaia languages file. We already do this for desktop builds, but we need to do it here, too.
Attachment #688856 - Flags: review?(aki)
Attachment #688856 - Flags: review?(aki) → review+
Attachment #688856 - Flags: checked-in+
Can't land this until bug 817170 is fixed. Rail, flagging you too, since you caught some problems the last time I tried to land this.
Attachment #688862 - Flags: review?(rail)
Attachment #688862 - Flags: review?(aki)
Comment on attachment 688862 [details] [diff] [review]
re-enable multilocale on all device builds

Happy to hear bug 817170 is on its way to being fixed.
Attachment #688862 - Flags: review?(aki) → review+
Comment on attachment 688862 [details] [diff] [review]
re-enable multilocale on all device builds

It looks like Rail clicked the checked-in flag instead of review -- adjusting.
Attachment #688862 - Flags: review?(rail)
Attachment #688862 - Flags: review+
Attachment #688862 - Flags: checked-in+
Comment on attachment 688862 [details] [diff] [review]
re-enable multilocale on all device builds

Landed. Reconfig to enable these will happen shortly.
Attachment #688862 - Flags: checked-in+
Forgot to set gaia languages file here, whoops.
Attachment #689270 - Flags: review?(aki)
Attachment #689270 - Flags: review?(aki) → review+
Attachment #689270 - Flags: checked-in+
In production since 9:00PT
Status update: Multilocale Gaia is now enabled for the RelEng Unagi builds (all branches). I flashed my own phone with it and it worked fine modulo other known bugs.

Still left to do here is enabling multilocale Gecko after dependent bugs are fixed.
Depends on: 819979
Depends on: 820160
Need equivalent patches for aurora/beta
Attachment #690951 - Flags: review?(catlee)
Attachment #690952 - Flags: review?(catlee)
Needed for gecko multilocale.
Attachment #690953 - Flags: review?(catlee)
Attachment #690953 - Flags: approval-mozilla-beta?
Comment on attachment 690952 [details] [diff] [review]
aurora gecko l10n config

Needed for gecko multilocale
Attachment #690952 - Flags: approval-mozilla-aurora?
Attached patch pass --gecko-languages-file (obsolete) — Splinter Review
As advertised. We don't need anything else here, because the gecko l10n root will be defined in the gecko config.
Attachment #691055 - Flags: review?(aki)
Same model as Gaia -- get locales file from command line, get repo root from gecko config.
Attachment #691063 - Flags: feedback?(aki)
Attachment #691055 - Flags: review?(aki) → review+
Comment on attachment 691063 [details] [diff] [review]
device multilocale support; builds but haven't tested on device

Awesome.
I think I could r+ this patch as well, but it looks like you want to get this tested first?

Sorry for the LocalesMixin super() weirdness.  I still haven't figured out how to do that well.
Attachment #691063 - Flags: feedback?(aki) → feedback+
Status update:
I'm currently doing initial test builds for otoro with multilocale gecko enabled. I will be handing these to QA to look at, and if things look good I'll be enabling multilocale gecko for devices soon afterwards. I need the Gecko patches from both bugs 812833 and 819979 to be reviewed and approved before enabling on mozilla-central. I also need the gecko patches from bug 796051 and 817197 uplifted to aurora/beta/b2g18 before I can enable multilocale gecko for devices on those branches.
Attachment #690952 - Flags: review?(catlee) → review+
Attachment #690953 - Flags: review?(catlee) → review+
Attachment #690951 - Flags: review?(catlee) → review+
Comment on attachment 690951 [details] [diff] [review]
set gecko l10n root in gecko configs

Landed on mozilla-central: https://hg.mozilla.org/mozilla-central/rev/aed7413b2efc
Attachment #690951 - Flags: checked-in+
Attachment #690952 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Attachment #690953 - Flags: approval-mozilla-beta? → approval-mozilla-b2g18+
Tested a build built with this on my unagi phone and it worked fine. I've asked QA to test on Otoro. Changes since the last patch:
* Add new required steps (checkout-gecko-l10n and checkout-compare-locales) to the configs.
* Add checkout-compare-locales step, because the Gecko build system requires it.
* Fix locales_files paths
* Set all the necessary variables to make the B2G and Gecko build systems do multilocale
Attachment #691055 - Attachment is obsolete: true
Attachment #691063 - Attachment is obsolete: true
Attachment #691468 - Flags: review?(aki)
Comment on attachment 691468 [details] [diff] [review]
device multilocale support, tested on unagi

you can get rid of checkout-compare-locales if you define

    l10n_repos: [{'repo': 'url/to/compare-locales', 'revision': ...}]

http://hg.mozilla.org/build/mozharness/file/09fe3cc6ad07/mozharness/mozilla/l10n/locales.py#l159

but this is fine.
Attachment #691468 - Flags: review?(aki) → review+
This patch includes the final bits we need for 812836, too. It enables gecko multilocale for desktop + device builds on central, aurora, and b2g18. We have to wait until bugs 796051 and 817197 are backported before we land. Axel says he's going to land those today.

I also want to wait for QA to get back to me on the Otoro build I sent them to test.

Changes since the last version:
* Fix gecko languages file paths
* Re-enable multilocale flag for aurora/b2g18
Attachment #691473 - Flags: review?(aki)
Attachment #691473 - Flags: review?(aki) → review+
Previous patch needed unbitrotting, and I had to remove a bunch of the gecko_language_file overriding that was happening.

Planning to land as soon as this is reviewed.
Attachment #691473 - Attachment is obsolete: true
Attachment #691811 - Flags: review?(catlee)
Attachment #691811 - Flags: review?(catlee) → review+
Comment on attachment 691811 [details] [diff] [review]
enable gecko multilocale everywhere

Reconfig for this happening shortly.
Attachment #691811 - Flags: checked-in+
Attachment #691468 - Flags: checked-in+
All RelEng device builds now have gecko and gaia multilocale enabled.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Summary: Set up nightly multilocale B2G unagi builds for dev testing → Use multilocale gecko and gaia for b2g device builds (unagi, otoro, panda)
Duplicate of this bug: 812831
Whiteboard: [status-b2g18:fixed]
Whiteboard: [status-b2g18:fixed]
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.