1410923 - Add locale fallbacking chains

Status: NEW

(Core :: Internationalization, enhancement, P3)

Description

[Zibi Braniecki [:zbraniecki][:gandalf]]

At the moment, the only fallbacking we do is from any locale to en-US.

It's hardcoded as:
 - locale we package fallback files for
 - locale we add to requested locales (bug 1410733)
 - locale we add to available locales (bug 1410731)

Instead, we should develop a dataset of locale fallback chains and then make it possible to retrieve it at runtime and build time.

My initial idea would be to:

 - get it as a .json file somewhere in `intl/locale/`
 - add python/mozbuild python command to retrieve a fallback for a given locale(s)
 - add LocaleService method to retrieve a fallback for a given locale(s)
 - switch the current uses to use the new API

The API could just do ['ab-CD', 'en-US'] unless an exception is listed in the new .json data structure.

Comment 1

[Zibi Braniecki [:zbraniecki][:gandalf]]

Axel, what do you think? In bug 1414390 I'm going to liberate us technically to store a list of locales as requested ones.

In this bug I'd like to consider, either in one bug, or split into two, a way to:

1) Add additional fallbacks to MOZ_CHROME_MULTILOCALE at build time
2) Add additional fallbacks to LocaleService::GetRequestedLocales at runtime

My current idea is sth like this:

a) in intl/locale/fallbacks.json we'd have sth like:

```
{
  "sr-RU": ["ru", "sr-SR"]
}
```

b) when, in the future, we get to build multilocale builds and someone builds:

MOZ_CHROME_MULTILOCALE="sr-RU, de, ru" ./mach package

we would look into the json file and extend the MOZ_CHROME_MULTILOCALE to "sr-RU, ru, sr-SR, de, en-US".

b) when someone is building a langpack with:

./mach build langpack-sr-RU

we would actually package ["sr-RU", "ru", "sr-SR", "en-US"]

c) at runtime, when someone calls GetRequestedLocales with `intl.locale.requested` set to "sr-RU,de,ru" we would return ["sr-RU", "ru", "sr-SR", "de", "en-US"]

Does it sound reasonable to you?

Comment 2

[Axel Hecht [:Pike]]

I'd comment on build-time next year, mostly because I'm on PTO during/after Austin, and the conversation there might impact what we do big-picture-wise.

We could work on runtime now, probably, but I'm not sure it's worth it yet.

For now, it'd probably be good to get the actual dataset of locales that need specific fallback, so that we can make technical decisions based on practical need.

Comment 3

[Zibi Braniecki [:zbraniecki][:gandalf]]

Flod, Delphine - is gathering a list of locale fallbacks something you'd be interested in tackling?

I'd be mostly curious about locales that wouldn't want a direct ["ab-CD", "en-US"] fallback. For example, `en-IN` or `en-AU` may want `en-GB`, `es-MX` would probably want `es` etc.

Comment 4

[Delphine Lebédel [:delphine - Use needinfo]]

I can help with this. I'll reach out and discuss what's the best way to gather this list.

Comment 5

[Delphine Lebédel [:delphine - Use needinfo]]

I was thinking a possible plan would be to reach out to locale leaders from each locale we have on Pontoon, and gather locale fallback preferences this way. I can set up a simple survey and send that out - recommending to the locale leader to reach out to the wider l10n community (when applicable) to give input as well before voting.

I can just use something like the list of managers Flod had gathered from Pudelo times to quickly send something out (I don't believe it's changed much since then - but I can make sure to compare and fill in the gaps/changes when needed).

@flod, @zibi: how does that sound to you?

Comment 6

[Delphine Lebédel [:delphine - Use needinfo]]

(hit send too soon)
For locales that we don't have on Pontoon (like en-IN or en-AU) I'm not quite sure though. Some seem "logical" and easy to guess, but I'm assuming it's not the case for all. I'll have to think that part through more :)
Is there a list of non-Pontoon locales that we know we're interested in getting this data for, maybe?

Comment 7

[Francesco Lodolo [:flod]]

The survey sounds like a good idea to me, but I need to get a better idea of the actual ask first.

This locale fallback chain is for UI language, correct? If that's the case, I think we should only cover locales that are in our builds, both as keys in the JSON, and as values for each locale. Otherwise, I have no idea how we can collect this data.

For example, I don't expect to have:

  "sr-RU": ["ru", "sr-SR"]

because we don't support sr-RU or sr-SR in our builds. I would expect something like this for Fulah:

  "ff": ["fr"]

Resulting in a response ["ff", "fr", "en-US"]. The default is the key itself, the last is en-US (implicit).

I would also like to be able to strip regions when not strictly necessary. If I ask for "sr-SR", would it fall back automatically to "sr" if it's in the list?