934196 - mach command for multilocale APK generation

Status: RESOLVED INCOMPLETE

(Firefox for Android Graveyard :: General, defect)

Description

[Richard Newman [:rnewman]]

Right now, building a multilocale APK for Fennec is a multi-step process involving checking out mozharness, *re-cloning the repo you want to build*, and hand-editing a config file:

https://wiki.mozilla.org/Mobile/Fennec/Android#Multilocale_builds

This seems needlessly difficult. There's no reason why I should have to clone my repo in order to make a new APK.

Is there a better component for this bug?

Comment 1

[Gregory Szorc [:gps]]

This falls under the realm of the Fennec build system.

Comment 2

[Mike Hommey [:glandium]]

Actually, the instructions are just limited to using mozharness, but you don't have to.

You can clone the languages you want, build --with-l10n-base (see the instructions for single locale builds) and add the following to your mozconfig:

   mk_add_options 'export MOZ_CHROME_MULTILOCALE="list of locales"'

And you're done. The latter is kind of awkward, but that can be made less awkward by adding AC_SUBST(MOZ_CHROME_MULTILOCALE) to configure.in, in which case you'd only need to add

   MOZ_CHROME_MULTILOCALE="list of locales"

Comment 3

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #2)
> And you're done.

Well, you still have to use compare-locales, though, and do something to have the locales generated (which i think would be the same as the last command listed for single locale builds, but with libs-$(AB_CD) instead of installers-$(AB_CD))

Comment 4

[Axel Hecht [:Pike]]

I never use mozharness for similar reasons as rnewman, I have my repos where I want them, and I usually like the additional control.

Here's a few things I do often, after configure and some make tier-base to get nsinstall etc:

make wget-en-US EN_US_BINARY_URL=ftp://....

make merge-% LOCALE_MERGEDIR=$PWD/merge-loc
make installers-% LOCALE_MERGEDIR=$PWD/merge-loc

for single locale builds and for multi-locale builds

make merge-% LOCALE_MERGEDIR=$PWD/merge-loc
make chrome-% LOCALE_MERGEDIR=$PWD/merge-loc

and then in the top-level build dir

make package MOZ_CHROME_MULTILOCALE="en-US list of locales" AB_CD=multi

Note that en-US needs to be part of the MOZ_CHROME_MULTILOCALE, but it's not used for chrome-%.

Getting all the damn repos is not a problem for me, as I have them anyway. Pitfalls often include finding the right EN_US_BINARY_URL, and making sure the MERGEDIR is right and absolute. Also, the multi package step is non-pretty, including the small mismatch between having en-US and not.

What I find is that when I actually have to do this, I often have to do it to find out why something's broken, and that's easier when I go through step-by-step.

And yes, multi uses chrome-% and not libs-%. The difference seems to be just xpi-stage vs dist.

Comment 5

[Richard Newman [:rnewman]]

(In reply to Mike Hommey [:glandium] from comment #2)

> Actually, the instructions are just limited to using mozharness, but you
> don't have to.

At the risk of tech-support-in-bug: I've added

mk_add_options 'export MOZ_CHROME_MULTILOCALE="en-US es-ES fr"'
ac_add_options --with-l10n-base=../l10n

to my .mozconfig.

fx-team$ ls ../l10n/es-ES
b2g		calendar	dom		extensions	mobile		other-licenses	services	toolkit
...

./mach build && ./mach package

Should I be expecting the built APK to now contain en-US, es-ES, and fr? As far as I can tell by unzipping and looking inside omni.ja, this is still an English-only build.

(It doesn't work with es_ES instead of es-ES, either, fwiw.)

Comment 6

[Axel Hecht [:Pike]]

You're missing a

cd objdir/mobile/android/locales
for loc in es-ES fr; do make merge-$loc LOCALE_MERGEDIR=$PWD/merge-$loc; make chrome-$loc LOCALE_MERGEDIR=$PWD/merge-$loc; done

between build and package.

PS: I glanced at some mach command providers, that's interesting science :-)

Comment 7

[Richard Newman [:rnewman]]

Slight amendment: needs L10NBASEDIR to run. Still working on making it actually produce a multilocale APK :P

for loc in es-ES; do make merge-$loc LOCALE_MERGEDIR=$PWD/merge-$loc; make chrome-$loc L10NBASEDIR=~/moz/hg/l10n LOCALE_MERGEDIR=$PWD/merge-$loc; done

Comment 8

[Richard Newman [:rnewman]]

OK, anyone know what's going on here?

$ find objdir-droid -name 'es-ES'
objdir-droid/dist/bin/chrome/es-ES
objdir-droid/dist/bin/chrome/es-ES/locale/es-ES
objdir-droid/mobile/locales/.deps/generated_es-ES/Users/rnewman/moz/hg/l10n/es-ES


That's the aftermath of building then running:

https://rnewman.pastebin.mozilla.org/3484355


./mach package produces interesting output like:

...mobile/android/installer/package-manifest:491: Missing file(s): bin/chrome/en-US es-ES
...mobile/android/installer/package-manifest:492: Missing file(s): bin/chrome/en-US es-ES.manifest

Comment 9

[Axel Hecht [:Pike]]

The merge-es-ES command doesn't produce the output I'd expect, which means that LOCALE_MERGEDIR isn't making it in, which may be related to the shell used, IIRC. For some

make FOO=BAR
does the same thing as for some others
FOO=BAR make
does, IIRC.

The merge-es-ES command should echo the same output as

compare-locales -m objdir-droid/mobile/android/locales/merge-es-ES mobile/android/locales/l10n.ini /Users/rnewman/moz/hg/l10n es-ES

Comment 10

[Richard Newman [:rnewman]]

By doubling up the LOCALE_MERGEDIR vars (zsh, thought it worked the same as bash, as opposed to csh), and boosting my .mozconfig section to

# l10n
mk_add_options 'export MOZ_CHROME_MULTILOCALE="en-US es-ES"'
mk_add_options 'export L10NBASEDIR="/Users/rnewman/moz/hg/l10n"'
ac_add_options --with-l10n-base=../l10n

I now get the correct output from the locale merge loop:

es-ES:
keys: 366
unchanged: 455
changed: 4544
obsolete: 20
missing: 20
90% of entries changed

What I don't get is correct packaging: 

  package-manifest:491: Missing file(s): bin/chrome/en-US es-ES

Comment 11

[Richard Newman [:rnewman]]

Oh, package-manifest.

--
[multilocale]
bin/chrome/en-US
bin/chrome/en-US.manifest
bin/chrome/en-US es-ES
bin/chrome/en-US es-ES.manifest
--

Comment 12

[Richard Newman [:rnewman]]

Yeah, that'll do it:

ifdef MOZ_CHROME_MULTILOCALE
        printf "\n[multilocale]\n" >> $@
        for LOCALE in en-US $(MOZ_CHROME_MULTILOCALE) ;\

MOZ_CHROME_MULTILOCALE better not be a quoted string inside mk_add_options. Which of course mine was. (Boy, I wish this were documented somewhere Google could find it!)

Comment 13

[Richard Newman [:rnewman]]

The following works.

--- .mozconfig
# l10n
mk_add_options 'export MOZ_CHROME_MULTILOCALE=en-US es-ES'
mk_add_options 'export L10NBASEDIR=/Users/rnewman/moz/hg/l10n'
ac_add_options --with-l10n-base=/Users/rnewman/moz/hg/l10n
---

From a clear dir:

./mach build
pushd objdir-droid/mobile/android/locales
for loc in es-ES; do LOCALE_MERGEDIR=$PWD/merge-$loc make merge-$loc LOCALE_MERGEDIR=$PWD/merge-$loc; make LOCALE_MERGEDIR=$PWD/merge-$loc chrome-$loc LOCALE_MERGEDIR=$PWD/merge-$loc; done
popd
./mach package


Next steps:

* Can we get the build and package steps to function without those mozconfig options? It seems weird that they should be there rather than arguments to the later stages. (Or, phrased differently, why should my mozconfig limit what I can put into my packaged output?)
* Adding a mach stage to install compare-locales along with the rest of our Python tooling. (We don't want to make developers fight with easy_install or pip.)
* Adding a mach stage that will do the compare-locales work.
* Augmenting the packaging step to do the right thing for a multilocale build -- e.g., naming the APK something other than 'en-US'.

Comment 14

[Richard Newman [:rnewman]]

Also, zomg this is so much more pleasant than working with multil10n.py. No awful rsyncing of object directories, for the most part, but also a sane patch workflow. Thanks to you both for the pointer!

Comment 15

[Mike Hommey [:glandium]]

(In reply to Richard Newman [:rnewman] from comment #13)
> * Can we get the build and package steps to function without those mozconfig
> options? It seems weird that they should be there rather than arguments to
> the later stages. (Or, phrased differently, why should my mozconfig limit
> what I can put into my packaged output?)

You don't have to. I thought it would be more convenient. You can pass them on the make command line. Mach just doesn't allow this...

> * Adding a mach stage to install compare-locales along with the rest of our
> Python tooling. (We don't want to make developers fight with easy_install or
> pip.)

Why don't we import compare-locales in the tree. That doesn't prevent from having a separate repo too (look at pymake, for example)

> * Adding a mach stage that will do the compare-locales work.
> * Augmenting the packaging step to do the right thing for a multilocale
> build -- e.g., naming the APK something other than 'en-US'.

You just need to set AB_CD=multi.

Comment 16

[Axel Hecht [:Pike]]

I think that having 

ac_add_options --with-l10n-base=/Users/rnewman/moz/hg/l10n

in one's mozconfig is very handy. I'm confused that you'd need it in the environment, too. Maybe our support for zsh users is really awful? It would surprise me, though. The MOZ_CHROME_MULTILOCALE should definitely just be a var during packaging.

I wonder if we could amend ./mach package with --multi-locale=es-ES,de,fr and make that do the right thing.

I'm not opposed to have compare-locales in the tree, I guess it'd have good fall-out to making the updates to the used copy more predictable and easier, if they just ride the trains. That said, no idea if the version would be fine for folks to take as is. It's code I wrote a long time ago, for the most part.

Comment 17

[Gregory Szorc [:gps]]

I'm all for having as much multilocale/l10n repack foo in the tree as possible. The inability for local developers to easily test these things has been a royal pain point for years. The desired end goal is for release automation to invoke a simple one-liner to perform all l10n foo. Same code paths locally and in automation.

Comment 18

[BMO Automation]

We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.