1358824 - [meta] Add ability to build multi-locale/multilingual Firefox

Status: NEW

(Firefox Build System :: General, enhancement)

Description

[Zibi Braniecki [:zbraniecki][:gandalf]]

One of the features that Fennec brought that we should extend to all Gecko builds is ability to generate multi-locale build for L10nRegistry.

I'd like to propose a config flag  "--locales" which would allow the user to define the list of locales to package.

```
./mach build --locales pl, de, fr
```

Additionally, the list of locales could be specified in the mozconfig via

```
ac_add_options --locales="pl, de, fr"
```

The result would be that the given language resources would be picked up from `--with-l10n-base` directory and packaged with the app.

Secondly, we'd need a runtime instruction for L10nRegistry that would register a FileSource for the app for the given path and the list of locales. For example:

```
const fs = new FileSource("app", ["pl", "de", "fr"], "resource://browser/localization/{locale}/");
L10nRegistry.addSource(fs);
```

That's not related to langpack building. I'll file a separate bug for that.

Comment 1

[Zibi Braniecki [:zbraniecki][:gandalf]]

:axel, can you guide me through how you'd like it to look like?

Comment 2

[Axel Hecht [:Pike]]

Justin, do you have thoughts on this already?

My personal take is probably that I want the list to be in the l10n build config, I should have more details on that after the weekend. Maybe something that can be overwritten at build time, and most easily so if we had a single mach command for both building locales and packaging?

Comment 3

[Zibi Braniecki [:zbraniecki][:gandalf]]

> My personal take is probably that I want the list to be in the l10n build config

I imagine that a build config can have those settings as well.

And then the fallback would look like this:

1) If passed as a build option, use it
2) If not, but build config available, use it
3) If not, go for a single-locale build using the default locale.

My biggest problem now is to find where in the code should we inject the file source.

Maybe we will end up with just a json file in a known location that can have a form like this:

{
  "app": {
    locales: ["pl", "de", "fr"],
    path: "resource://browser/localization/{locale}/"
  },
  "platform": {
    locales: ["pl", "de", "fr"],
    path: "resource:///gre/localization/{locale}/"
  },
}

?

I'd need then to read it the first time L10nRegistry is opened, but that would free us from the need to fire some JS/C++ code to register the packaged sources?

Comment 4

[Justin Wood (:Callek)]

(In reply to Axel Hecht [out-of-office -May 2nd][:Pike] from comment #2)
> Justin, do you have thoughts on this already?
> 
> My personal take is probably that I want the list to be in the l10n build
> config, I should have more details on that after the weekend. Maybe
> something that can be overwritten at build time, and most easily so if we
> had a single mach command for both building locales and packaging?

I'd love a bigger picture view on what and why here, before I offer too much in terms of thoughts. I wasn't aware there was a desire for MultiLocale in desktop in any way.

I certainly have some "what we do today" caveats to share, and other information about where I hope to be heading with this all.

Maybe once you (Pike) are back, us 3 (Gandalf, You and I) can have a meeting to go over this?

(I'm on PTO myself, May 8 through May 20)

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #3)
> > My personal take is probably that I want the list to be in the l10n build config
> 
> I imagine that a build config can have those settings as well.
> 

Roughly speaking, much of l10n automation is driven by mozharness right now, and I'd like to not add too many different ways to invoke stuff.

That said, getting rid of Mozharness and using ./mach is part of my end goal, but it gets pretty hard to disentangle until *after* we're no longer using Buildbot for L10n at all (any platform) so I've been band-aiding mozharness in the mean time.

There's also some issues with signed-off-locales and Beta/Release in terms of doing multi-locale jobs, right now the en-US build we ship comes directly from CI, for desktop. For Android we generate an entirely new build because of multi-locale l10n.

Comment 5

[(not currently active) Ted Mielczarek]

From a build system perspective, having a configure option like `--with-locales=ab-CD,wx-YZ` is very straightforward, and writing out a config file like you describe based on the value of the configure option would also be simple.

Comment 6

[Zibi Braniecki [:zbraniecki][:gandalf]]

> I'd love a bigger picture view on what and why here, before I offer too much in terms of thoughts. I wasn't aware there was a desire for MultiLocale in desktop in any way.

Generally speaking, my current refactor of the locale loading in Gecko is leading us in the direction of having two "methods" of getting locales into the app:

 - bundle locales in the app package
 - create langpack

The langpack route is not yet fully flushed out, but I imagine we'll end up with something like:

```
./mach build-langpack --locales=pl[,de,fr]
```

 - where most of the time the langpack will have just one locale, but in theory it would be possible to create a multi-lingual langpack.

Then, I imagine that we'd be slowly closing the gap between those two methods and eventually end up with the system in which every language is added as a langpack, but some langpacks can be shipped with the package.

There are currently two targets for the multi-lingual Firefox:
 - for Fennec we want to be able to ship langpacks as DLC to reduce the size of APK file
 - for Firefox Desktop we want to be able to ship a single Firefox (via Windows Store for example)

This bug is the first step - in order to be able to reliably test the runtime infrastructure I need to be able to produce a multi-lingual package.
The langpack via DLC bug is bug 1347803.

> Maybe once you (Pike) are back, us 3 (Gandalf, You and I) can have a meeting to go over this?

Sure, I'll sync with Axel and throw a meeting :)

Comment 7

[Axel Hecht [:Pike]]

Here's a thing I think we should make easy:

A developer hacks on a patch, and suddenly finds out that she should test a localized build.

Can a configure flag avoid recompiling the world in that case?

Comment 8

[Zibi Braniecki [:zbraniecki][:gandalf]]

I agree that rebuilding locales should be easy - similar maybe to the "faster" mode to avoid all the c++ pieces.

But for that particular use case, down the road, I hope we'll use pseudo locales :)

Comment 9

[Zibi Braniecki [:zbraniecki][:gandalf]]

We met with :pike and :callek and decided to plan for the three commands:

1) ./mach build --locales pl,de,fr

This will allow us to build Firefox with the three locales packaged in.

2) ./mach build locales

(analogous to "./mach build faster") This will allow us to rebuild Firefox doing only the locale repackaging

3) ./mach build-langpack --locales pl,de,fr

This will allow us to build a langpack(s) for given locale(s).


:glandium, in bug 1361860 you indicated that we can't get a subcommand for build. Can you advice how to get to what we're aiming for here?

Comment 10

[Gregory Szorc [:gps]]

I'm not sure you'll be able to use a mach sub-command proper because `mach build` currently takes a positional argument and that seems incompatible with how mach parses sub-commands. So you'll either need to implement a mach feature to do "fall back" sub-command dispatch or just one-off those build targets in the `mach build` handler.

Comment 11

[Zibi Braniecki [:zbraniecki][:gandalf]]

With bug 1362496 fixed and bug 1362617 in review I'm able to produce multi-lingual Firefox!

Unfortunately, packaging step errors with duplicates: https://pastebin.mozilla.org/9026097

:Pike, who's the right person to talk to about it?

Comment 12

[Marc Bauer]

With Windows Enterprise we have 42 languages on every desktop and I'm looking for a MUI installer for about 10 years. I need to install all languages and the Windows UI (not the OS language) language need to be used than. Is automatic language switching now possible?

Since you can now build this MUI installer, where can I download it?

Chrome has this MUI installers for years.

Comment 13

[Francesco Lodolo [:flod]]

(In reply to Marc Bauer from comment #12)
> Since you can now build this MUI installer, where can I download it?

This bug is still open, there's no multilocale installer at the moment. There will be one, but there's also no explicit timeline at the moment.

P.S. please don't add the same comment to multiple bugs ;-)

Comment 14

[Marc Bauer]

Please add a roadmap with timeline - before all people are on Chrome.

Comment 15

[Marc Bauer]

One year passed away and we still have no MUI version. Can you put a timeline on it, please? Otherwise I will uninstall Firefox soon and force 500 users to Chrome only.

Comment 16

[Adam Wood]

I've been looking into building multi-locale Firefox and have managed to get it working for en-US and one other locale. When attempting to build with more than one additional locale I'm still encountering some issues (highlighted at the end).

I've been using the below mach command:

1. ./mach package-multi-locales --locales en-US de (now works)
2. ./mach package-multi-locales --locales en-US de fr (doesn't work)

Once the multi-locale package begins after the build stages the below errors will be received (building on Mac, so platform specific and not main focus):
Error: $SRCDIR/browser/installer/package-manifest.in:44: Missing file(s): Nightly.app/Contents/Resources/multi.lproj/*

As the app name remains consistent regardless of locale, the following lines can be added to browser/installer/package-manifest.mk starting at line 44:

#if __LPROJ_ROOT__ == en
@RESPATH@/@LPROJ_ROOT@.lproj/*
#endif

This hack removes the requirement for a locale specific .lproj folder to exist, although the ideal solution would be to make sure that the folders are correctly created for all specified locales. This does, however, resolve the error from above which allows the packaging to get slightly further along although it will now fail with the below:

RuntimeError: File "necko.properties" not found in $objdir/browser/locales/merge-dir/multi/netwerk

This is because AB_CD is set as multi so the packaging is looking for merge-dir/multi which doesn't exist, rather than merge-dir/<locale> (which does exist for all supplied locales). To resolve this I implemented the below in python/mozbuild/mozbuild/mach_commands.py, starting at ~line 1591:

for locale in locales:  
    if locale == 'en-US':
        self.log(logging.INFO, 'package-multi-locale', {'locale': locale},
                  'Skipping default locale {locale}')
        continue       
            
    self.log(logging.INFO, 'package-multi-locale', {'locale': locale},
              'Invoking multi-locale `mach package`, {locale}')
    self._run_make(
        directory=self.topobjdir,
        target=['package', 'AB_CD={}'.format(locale)],
        append_env=append_env,
        pass_thru=True,
        ensure_exit_code=True)

This then allows a multi (2) locale build of Firefox to successfully be packaged.

Current Issues
Where the above begins to fail is when attempting to build a true multi locale package, with more than one additional locale. The build stage will still complete “successfully” however only the merge-dir/<locale> will be generated successfully, with the relevant chrome folders in the objdir not being generated for any aside from the first additional locale. This produces the below errors:

Error: $OBJDIR/browser/installer/locale-manifest.in:4: Missing file(s): Nightly.app/Contents/Resources/browser/chrome/fr
Error: $OBJDIR/browser/installer/locale-manifest.in:5: Missing file(s): Nightly.app/Contents/Resources/browser/chrome/fr.manifest
Error: $OBJDIR/browser/installer/locale-manifest.in:6: Missing file(s): Nightly.app/Contents/Resources/chrome/fr
Error: $OBJDIR/browser/installer/locale-manifest.in:7: Missing file(s): Nightly.app/Contents/Resources/chrome/fr.manifest

Which is the same as what happens for one additional locale before adding "GRADLE_INVOKED_WITHIN_MACH_BUILD": "1" to append_env in mach_commands.py. So my question here is, why are the relevant chrome and browser/chrome files generated for the first additonal locale, but not for the second? The browser/locales/merge-dir/<locale> files are generated correctly for all additional locales.
Could you assist in where would be a good place to do some more investigation for this? Where is the generation of the chrome and browser/chrome files controlled (assuming one or more of the jar.mn) and why would this only work for the first ./mach build chrome-<locale> called from ./mach package-multi-locales and not for any subsequent one's?

Comment 17

[Nick Alexander :nalexander [he/him]]

Adam, this is complicated and my memory is somewhat hazy. But let me try to help as I can.

I started to say that you will need to look at and understand the differences between Desktop's packaging and Android's packaging as well as some subtleties around the multi locale, but then I realized that I couldn't really follow-up without digging into the code. Rather than give too much explanation, I will suggest the following patch (which you might test) which works for me. The key issue you saw was not running mach build browser/app/tools or its equivalent: that populates the Contents/Resources directory. I have not verified that the resulting package has a healthy omnijar, although I expect it does. I have verified that in the Browser Console, we have

Cc["@mozilla.org/intl/localeservice;1"].getService(Ci.mozILocaleService).availableLocales == [ "de", "en-US", "fr" ]

and that

Services.prefs.setCharPref("intl.locale.requested", "fr");

does in fact change the localization of the browser. (Also, that "de" succeeds.) Play with this and see if I have missed anything more here?

diff --git a/python/mozbuild/mozbuild/mach_commands.py b/python/mozbuild/mozbuild/mach_commands.py
--- a/python/mozbuild/mozbuild/mach_commands.py
+++ b/python/mozbuild/mozbuild/mach_commands.py
@@ -2015,15 +2015,34 @@ class L10NCommands(MachCommandBase):
                 cwd=mozpath.join(self.topsrcdir),
             )
 
+        if self.substs["MOZ_BUILD_APP"] == "browser":
+            self.log(
+                logging.INFO,
+                "package-multi-locale",
+                {},
+                "Repackaging browser",
+            )
+            self._run_make(
+                directory=mozpath.join(self.topobjdir, "browser", "app"),
+                target=["tools"],
+                append_env=append_env,
+                pass_thru=True,
+                ensure_exit_code=True,
+            )
+
         self.log(
             logging.INFO,
             "package-multi-locale",
             {},
             "Invoking multi-locale `mach package`",
         )
+        target = ["package"]
+        if self.substs["MOZ_BUILD_APP"] == "mobile/android":
+            target.append("AB_CD=multi")
+
         self._run_make(
             directory=self.topobjdir,
-            target=["package", "AB_CD=multi"],
+            target=target,
             append_env=append_env,
             pass_thru=True,
             ensure_exit_code=True,

Comment 18

[Adam Wood]

Thanks so much for the explanation and the patch here Nick! Confirming that I've tested this and all seems to be working accordingly.

Comment 19

[Nick Alexander :nalexander [he/him]]

(In reply to Zibi Braniecki [:zbraniecki][:gandalf] from comment #8)

I agree that rebuilding locales should be easy - similar maybe to the
"faster" mode to avoid all the c++ pieces.

But for that particular use case, down the road, I hope we'll use pseudo
locales :)

Zibi: could you say a few words about what you were thinking here?

Comment 20

[Nick Alexander :nalexander [he/him]]

(In reply to Nick Alexander :nalexander [he/him] from comment #19)

(In reply to Zibi Braniecki [:zbraniecki][:gandalf] from comment #8)

I agree that rebuilding locales should be easy - similar maybe to the
"faster" mode to avoid all the c++ pieces.

But for that particular use case, down the road, I hope we'll use pseudo
locales :)

Zibi: could you say a few words about what you were thinking here?

Rats: NI to Zibi.

Comment 21

[Zibi Braniecki [:zbraniecki][:gandalf]]

Zibi: could you say a few words about what you were thinking here?

Did you just ask me willy-nilly what was I thinking 5 years ago? Ha! Bold! :)

My guess is that I was thinking that the use case of building Firefox with selected locales in order to test UI in those locales should be instead supplied by pseudo-locales.
And with repacks going away, the use case of "Give me Firefox with locale X" should be supplied by langpacks.
And the use case of "build Firefox with 4 locales for Switzerland" should be supplied by langpacks as well.

So, with repacks going away, the command of ./mach package --locales=de-CH,fr-CH,en-CH,it-CH should bundle four langpacks with Firefox rather than four locales in a repack.

Comment 22

[Nick Alexander :nalexander [he/him]]

(In reply to Zibi Braniecki [:zbraniecki][:gandalf] from comment #21)

Zibi: could you say a few words about what you were thinking here?

Did you just ask me willy-nilly what was I thinking 5 years ago? Ha! Bold! :)

I'm nothing if not optimistic :)

My guess is that I was thinking that the use case of building Firefox with selected locales in order to test UI in those locales should be instead supplied by pseudo-locales.

Ah, I follow you. Yes, we should use pseudo-locales for testing things in-product, absolutely.

And with repacks going away, the use case of "Give me Firefox with locale X" should be supplied by langpacks.
And the use case of "build Firefox with 4 locales for Switzerland" should be supplied by langpacks as well.

So, with repacks going away, the command of ./mach package --locales=de-CH,fr-CH,en-CH,it-CH should bundle four langpacks with Firefox rather than four locales in a repack.

We'll see where I get to in this direction.

Thanks!

Comment 23

[Nick Alexander :nalexander [he/him]]

I'm actively working on parts of this, so I'm going to make this a meta and break out some subtickets.