1365433 - Decide on how to handle localization "packages"

Status: NEW

(Core :: Internationalization, enhancement)

Description

[Zibi Braniecki [:zbraniecki][:gandalf]]

In several places we encountered a case for introducing a virtual "package" that contains a certain set of localization resources and potentially uses separate localization strategy.

Three such examples:

1) Firefox in "de", but DevTools in "en-US"

In this scenario, we'd like to easily be able to allow localizers to produce Firefox localization without having to localize DevTools.
What's more, users may want to use Firefox in their select locale, but keep the DevTools in en-US.

Those two needs could be fulfilled if we decide to separate out "devtools" package from firefox package. This way any language pack may contain resources just for firefox, or for firefox and devtools.

2) System extensions

We have a bunch of extensions that we ship with the product, called "system extensions". With the division into packages, we could ship a langpack that has strings for ["firefox", "screenshots", "minvid", "pocket"] and if any of the extensions is moved out of firefox, this package would be transitioned into an extension. This would also allow localizers to localize "firefox", but not "minvid" for example.

3) Extensions

We'd like to think on how the extensions localization will work using our L10n API. If we had those virtual packages, then any langpack could provide translations for any selection of extensions, down to a langpack-per-extension.

This taps into:
 - bug 1280690 - Determine how to slice Firefox into user-relevant things
 - bug 1280906 - Support "partial things" of a product in compare-locales

and could be added as an extension to L10nRegistry and new langpack model.

Comment 1

[Zibi Braniecki [:zbraniecki][:gandalf]]

:kmag - you indicated that such virtual package system could be useful for WebExtensions localization (if they wanted to use our l10n system instead of google's)
:stas, :flod, :pike - we talked about ideas around this for a while. I think it would be good to crystallize our vision here so that L10nRegistry and the new langpack model can be ready for that.

Comment 2

[Axel Hecht [:Pike]]

Most of these scenarios are going to be dealt with by project configuration in the near future.

That's a project I'm driving as part of x-channel l10n.

Basically, it'll give our ecosystem a place to switch off localization for a particular project, or a particular file in a project.

I think the main point of "package" is to enable language selection by the user independent of the general language selection.

As devtools is an example, the fact that it's moving to its own extension might support the idea that we just need to resolve this issue per-extension..

Comment 3

[Zibi Braniecki [:zbraniecki][:gandalf]]

> As devtools is an example, the fact that it's moving to its own extension might support the idea that we just need to resolve this issue per-extension..

I agree.

The idea I'm trying to verify is that once this happens, we'll want to indicate that a given language pack has resources for multiple "packages", for example: ["firefox", "devtools"].
Or would you prefer to install two langpacks - one for firefox, one for devtools?

Comment 4

[Axel Hecht [:Pike]]

I'm trying to think about this from a POV of how we can shield web extensions from firefox, and vice versa. Which is what devtools is going to end up being eventually.

In that scenario, I wonder if we should prevent a web extension from accessing firefox l10n resources. Might be the right thing to do.

I'd defer to Kris on whether we should do add-ons for add-ons here, or if doing some sort of in-add-on DLC is better, or something completely different.

... and yes, I know that I've said that packages are a thing, but with devtools moving to a proper add-on, the scenarios change quite a bit.

Comment 5

[Francesco Lodolo [:flod]]

I feel like I have more questions than answers at the moment.

I'm looking at bug 1280690 and thinking about the Web Console. I can totally see a developer using Firefox in a local language, not necessarily a minority one, and needing to google for the English error message because the translated one is not clear enough. That doesn't mean that they want the whole browser in English.

That's part of having a smart language selection for portions of Firefox. Let's say the "Firefox" language pack is made of "browser-ui"+"console-messages", we would switch only the latter, so virtual packages would be useful (necessary?). 

As long as the process to install a language pack is seamless for the user, I don't really care if it's a monolithic language pack or distinct language packs. The latter would probably save bandwidth (I want devtools in French, not the whole system), unless we decide to download all French language packs once you add French to your content-locales list.

How would you solve the problem of creating a monolithic language pack when the translations are scattered all over the place? To follow your example:
"firefox": hg
"screenshots": GitHub repo1
"minvid": GitHub repo2
"pocket":  GitHub repo3

Regarding accessing l10n resources from extensions: we had that request recently, for the Containers experiment on Test Pilot (in the end they went for en-US only). I don't know if there have been such requests in the past, so if it's worth thinking more about it before shutting down that door.

Comment 6

[Zibi Braniecki [:zbraniecki][:gandalf]]

> How would you solve the problem of creating a monolithic language pack when the translations are scattered all over the place?

That's a build system question. I believe we are capable of collecting data from multiple repositories.

What I'd like to resolve here is how to handle a language pack (or a languages bundles in the app) for when the packaged bundle is Firefox, and DevTools and Screenshots and MinVid and Pocket.

At the moment, L10nRegistry limits us to think about resources for all of them as a single "package" of data and that puts no boundaries (so devtools or minvid can use firefox resources - which is bad).
When you'll want to take such extension and package it as a proper extension removing it from Firefox, you'll have to look through all the resources it uses and move it away.
If we had such virtual "packages" we could make sure that devtools only access devtools resources, and firefox only accesses firefox resources.

Also, I'd like to get a vision for how we'll localize WebExtensions using Mozilla API. And that I believe is on :kris :)

Comment 7

[Kris Maglione [:kmag]]

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #0)
> 3) Extensions
> 
> We'd like to think on how the extensions localization will work using our
> L10n API. If we had those virtual packages, then any langpack could provide
> translations for any selection of extensions, down to a
> langpack-per-extension.

This will require some thought...

We already allow this for legacy extensions, by allowing any extension to require locale and theme packages for packages created by other extensions, and I know that it's had some usage over the years, but it would be interesting to see just how much.

I'd definitely be interested in trying to use our new localization architecture for WebExtensions. And I'd also love to allow users to choose a different locale for a particular extension than they do for the main UI (developers, for example, tend to prefer English for developer-oriented extensions even if they use their native language elsewhere... and some extensions tend to have pretty marginal translations). But we'd have to discuss how it would work from a technical level.

WebExtension localization works like this:

- The extension provides a _locales directory in the top-level of their extension, with one sub-directory containing a messages.json file that defines the messages for that extension.

- If the extension is localized, the extension manifest must provide a "default_locale" property, which defines the default locale if there is no good match with the browser locale, and decides the fallback locale for messages which are missing from the selected locale.

- At the start of the session, we choose the locale of the extension based on the currently selected browser locale, and load both that locale and the default locale. We do some preprocessing of the messages files and store them in a cache so that they're readily available at startup during the next session, so we should be able to mangle them into any format that our locale architecture expects, at least in theory.

- When a string is needed, we check the messages for the selected locale, followed by the default locale, followed by a set of built-in message codes. Ideally, I'd like to check more fallback locales based on the set of requested browser locales, so I'd be open to changing this behavior somewhat to match what platform code does.

- The localization API also allows retrieving the current browser UI locale, but this only returns one locale, not a list. If we decide to allow choosing extension-specific locales, we might need to decide whether to handle this differently in those cases.

If we wanted to allow extending the set of locales available for a given extension, it would probably make sense to allow another extension to specify a sub-directory containing messages for that extension, in a similar way to how we'll specify browser locales in whatever we do for new-style langpack extensions.