1331721 - l20n should be able to fallback to the closest locale ignoring the region identifier

Status: RESOLVED FIXED

(L20n :: General, defect)

Description

[Francesco Lodolo [:flod]]

We've already seen a few reports on Test Pilot. 

For example: accept-language is set to "en-GB,en,nl", en-GB is not available, we only have en-US. Instead of displaying en-US because of 'en', the website displays 'nl' (locale is available).

One other case is "de-DE" vs "de", website falls back to en-US, while it should be smarter to display 'de'
https://github.com/mozilla/testpilot/issues/2017

Comment 1

[Zibi Braniecki [:zbraniecki][:gandalf]]

So, I think that climbing up from "en" to "en-US" is unprecedented in language negotiation.

Language negotiation usually goes from more specific to more generic like this:

1) user requests es-MX-Latn
2) check if es-MX-Latn exists
3) if not, trim last bit and check for es-MX
4) if not, trim last bit and check for es

It's a bit more complex than that and ECMA402 has a draft proposal for Intl.getParentLocales to handle edge cases, but generally speaking if you have one English, you should name it "en". Then if you have a specialized variant that diverges, name it "en-AU". And so on.

So, for your particular request, wouldn't it make sense to switch en-US to en since this is the en you want to serve to all users who request any en that doesn't have its own specialized variant?

Comment 2

[Francesco Lodolo [:flod]]

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #1)
> So, for your particular request, wouldn't it make sense to switch en-US to
> en since this is the en you want to serve to all users who request any en
> that doesn't have its own specialized variant?

We would need to check with Matjaz that it doesn't break anything in Pontoon, and then remove the hack in Test Pilot that creates 'en' at build time (if devs agree).

Still, I think we should do something smarter, and that includes checking if any "en-*" is available in case "en" was requested but is not available.

Comment 3

[Axel Hecht [:Pike]]

There's https://developer.android.com/guide/topics/resources/multilingual-support.html and https://blog.golang.org/matchlang to just list a couple of examples where language negotiation is trying to do what humans expect.

Comment 4

[Zibi Braniecki [:zbraniecki][:gandalf]]

So, LDML defines some CLDR data for the guesswork - http://unicode.org/reports/tr35/tr35.html#LanguageMatching

If we reached out to use likelySubtags, and full language matching with matching percentages, we'd likely end up supporting en-US based on 'en'.

But, as Android docs you linked state - it's significantly better to keep 'en' for all en's, and keep en-* for diverging variants, than to keep guessing which en should be returned among en-US, en-AU, en-AF, en-CA, en-GB etc.

There are enough pitfalls when looking for parent locales, I believe there are even more when looking for children locales.

Comment 5

[Wil Clouser [:clouserw]]

We have a bug filed for this in Test Pilot (https://github.com/mozilla/testpilot/issues/2017) and are working on prioritizing it.  Is this bug scheduled to be fixed soon or should we work on making a patch on our side?

Comment 6

[Staś Małolepszy :stas]

The current language negotiation algorithm in l20n.js is very simple and doesn't do anything beyond exact-matching the language tags. This is an artifact from the Firefox OS days where we knew exactly what the language codes available to the user were.

On the web, things aren't this easy and we'll need a better algorithm to support many different variants and permutations of language settings users may have. There already exists a better algorithm and it's defined by ECMA 402: https://tc39.github.io/ecma402/#sec-lookupsupportedlocales

I'll submit a pull request to switch to using this algorithm shortly.  I've already verified that it fixes the issue reported by the TestPilot team.  It does come at a price of a larger size of the library: +17 KB not minified.

Comment 7

[Staś Małolepszy :stas]

Attachment: Pull request

Comment 8

[Staś Małolepszy :stas]

https://github.com/l20n/l20n.js/commit/c3e35c497b5168270cc629f0ee6996cb34033cb1