Add ability to build multilingual Firefox

NEW
Unassigned

Status

2 years ago
2 months ago

People

(Reporter: gandalf, Unassigned, NeedInfo)

Tracking

(Depends on: 2 bugs)

Firefox Tracking Flags

(Not tracked)

Details

One of the features that Fennec brought that we should extend to all Gecko builds is ability to generate multi-locale build for L10nRegistry.

I'd like to propose a config flag  "--locales" which would allow the user to define the list of locales to package.

```
./mach build --locales pl, de, fr
```

Additionally, the list of locales could be specified in the mozconfig via

```
ac_add_options --locales="pl, de, fr"
```

The result would be that the given language resources would be picked up from `--with-l10n-base` directory and packaged with the app.

Secondly, we'd need a runtime instruction for L10nRegistry that would register a FileSource for the app for the given path and the list of locales. For example:

```
const fs = new FileSource("app", ["pl", "de", "fr"], "resource://browser/localization/{locale}/");
L10nRegistry.addSource(fs);
```

That's not related to langpack building. I'll file a separate bug for that.
(Reporter)

Comment 1

2 years ago
:axel, can you guide me through how you'd like it to look like?
Flags: needinfo?(l10n)
(Reporter)

Updated

2 years ago
Blocks: 1347801
Justin, do you have thoughts on this already?

My personal take is probably that I want the list to be in the l10n build config, I should have more details on that after the weekend. Maybe something that can be overwritten at build time, and most easily so if we had a single mach command for both building locales and packaging?
Flags: needinfo?(bugspam.Callek)
(Reporter)

Comment 3

2 years ago
> My personal take is probably that I want the list to be in the l10n build config

I imagine that a build config can have those settings as well.

And then the fallback would look like this:

1) If passed as a build option, use it
2) If not, but build config available, use it
3) If not, go for a single-locale build using the default locale.

My biggest problem now is to find where in the code should we inject the file source.

Maybe we will end up with just a json file in a known location that can have a form like this:

{
  "app": {
    locales: ["pl", "de", "fr"],
    path: "resource://browser/localization/{locale}/"
  },
  "platform": {
    locales: ["pl", "de", "fr"],
    path: "resource:///gre/localization/{locale}/"
  },
}

?

I'd need then to read it the first time L10nRegistry is opened, but that would free us from the need to fire some JS/C++ code to register the packaged sources?
(In reply to Axel Hecht [out-of-office -May 2nd][:Pike] from comment #2)
> Justin, do you have thoughts on this already?
> 
> My personal take is probably that I want the list to be in the l10n build
> config, I should have more details on that after the weekend. Maybe
> something that can be overwritten at build time, and most easily so if we
> had a single mach command for both building locales and packaging?

I'd love a bigger picture view on what and why here, before I offer too much in terms of thoughts. I wasn't aware there was a desire for MultiLocale in desktop in any way.

I certainly have some "what we do today" caveats to share, and other information about where I hope to be heading with this all.

Maybe once you (Pike) are back, us 3 (Gandalf, You and I) can have a meeting to go over this?

(I'm on PTO myself, May 8 through May 20)

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #3)
> > My personal take is probably that I want the list to be in the l10n build config
> 
> I imagine that a build config can have those settings as well.
> 

Roughly speaking, much of l10n automation is driven by mozharness right now, and I'd like to not add too many different ways to invoke stuff.

That said, getting rid of Mozharness and using ./mach is part of my end goal, but it gets pretty hard to disentangle until *after* we're no longer using Buildbot for L10n at all (any platform) so I've been band-aiding mozharness in the mean time.

There's also some issues with signed-off-locales and Beta/Release in terms of doing multi-locale jobs, right now the en-US build we ship comes directly from CI, for desktop. For Android we generate an entirely new build because of multi-locale l10n.
Flags: needinfo?(bugspam.Callek)
From a build system perspective, having a configure option like `--with-locales=ab-CD,wx-YZ` is very straightforward, and writing out a config file like you describe based on the value of the configure option would also be simple.
(Reporter)

Comment 6

2 years ago
> I'd love a bigger picture view on what and why here, before I offer too much in terms of thoughts. I wasn't aware there was a desire for MultiLocale in desktop in any way.

Generally speaking, my current refactor of the locale loading in Gecko is leading us in the direction of having two "methods" of getting locales into the app:

 - bundle locales in the app package
 - create langpack

The langpack route is not yet fully flushed out, but I imagine we'll end up with something like:

```
./mach build-langpack --locales=pl[,de,fr]
```

 - where most of the time the langpack will have just one locale, but in theory it would be possible to create a multi-lingual langpack.

Then, I imagine that we'd be slowly closing the gap between those two methods and eventually end up with the system in which every language is added as a langpack, but some langpacks can be shipped with the package.

There are currently two targets for the multi-lingual Firefox:
 - for Fennec we want to be able to ship langpacks as DLC to reduce the size of APK file
 - for Firefox Desktop we want to be able to ship a single Firefox (via Windows Store for example)

This bug is the first step - in order to be able to reliably test the runtime infrastructure I need to be able to produce a multi-lingual package.
The langpack via DLC bug is bug 1347803.

> Maybe once you (Pike) are back, us 3 (Gandalf, You and I) can have a meeting to go over this?

Sure, I'll sync with Axel and throw a meeting :)
Here's a thing I think we should make easy:

A developer hacks on a patch, and suddenly finds out that she should test a localized build.

Can a configure flag avoid recompiling the world in that case?
(Reporter)

Comment 8

2 years ago
I agree that rebuilding locales should be easy - similar maybe to the "faster" mode to avoid all the c++ pieces.

But for that particular use case, down the road, I hope we'll use pseudo locales :)

Updated

2 years ago
Flags: needinfo?(l10n)
(Reporter)

Updated

2 years ago
Depends on: 1362496
(Reporter)

Comment 9

2 years ago
We met with :pike and :callek and decided to plan for the three commands:

1) ./mach build --locales pl,de,fr

This will allow us to build Firefox with the three locales packaged in.

2) ./mach build locales

(analogous to "./mach build faster") This will allow us to rebuild Firefox doing only the locale repackaging

3) ./mach build-langpack --locales pl,de,fr

This will allow us to build a langpack(s) for given locale(s).


:glandium, in bug 1361860 you indicated that we can't get a subcommand for build. Can you advice how to get to what we're aiming for here?
Flags: needinfo?(mh+mozilla)
(Reporter)

Updated

2 years ago
Depends on: 1362617
I'm not sure you'll be able to use a mach sub-command proper because `mach build` currently takes a positional argument and that seems incompatible with how mach parses sub-commands. So you'll either need to implement a mach feature to do "fall back" sub-command dispatch or just one-off those build targets in the `mach build` handler.
(Reporter)

Updated

2 years ago
Blocks: 1365426
No longer blocks: 1347801
(Reporter)

Comment 11

2 years ago
With bug 1362496 fixed and bug 1362617 in review I'm able to produce multi-lingual Firefox!

Unfortunately, packaging step errors with duplicates: https://pastebin.mozilla.org/9026097

:Pike, who's the right person to talk to about it?
Flags: needinfo?(l10n)
(Reporter)

Updated

2 years ago
Depends on: 1377543

Updated

2 years ago
Flags: needinfo?(l10n)

Updated

2 years ago
Depends on: 1390152
(Reporter)

Updated

a year ago
No longer blocks: 1365426

Comment 12

a year ago
With Windows Enterprise we have 42 languages on every desktop and I'm looking for a MUI installer for about 10 years. I need to install all languages and the Windows UI (not the OS language) language need to be used than. Is automatic language switching now possible?

Since you can now build this MUI installer, where can I download it?

Chrome has this MUI installers for years.
(In reply to Marc Bauer from comment #12)
> Since you can now build this MUI installer, where can I download it?

This bug is still open, there's no multilocale installer at the moment. There will be one, but there's also no explicit timeline at the moment.

P.S. please don't add the same comment to multiple bugs ;-)
Comment hidden (advocacy)

Updated

a year ago
Product: Core → Firefox Build System
(Reporter)

Updated

a year ago
Depends on: 356184

Comment 15

2 months ago
One year passed away and we still have no MUI version. Can you put a timeline on it, please? Otherwise I will uninstall Firefox soon and force 500 users to Chrome only.
You need to log in before you can comment on or make changes to this bug.