Locales are wrongly mapped in Fennec, causing an incorrect list of search engines to be displayed

RESOLVED FIXED in Firefox 56

Status

()

Firefox for Android
Settings and Preferences
RESOLVED FIXED
4 months ago
3 months ago

People

(Reporter: Grisha, Assigned: gandalf)

Tracking

(Blocks: 1 bug, {regression, regressionwindow-wanted})

unspecified
Firefox 57
regression, regressionwindow-wanted
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(fennec+, firefox55 wontfix, firefox56blocking fixed, firefox57 fixed)

Details

(Whiteboard: [FNC][SPT57.3][MVP])

(Reporter)

Description

4 months ago
After installing current 56 beta, I only see three default search engines in Settings:
- Google (default)
- Twitter
- Wikipedia

What happened with to the rest?
(Reporter)

Updated

4 months ago
See Also: → bug 1389710
(Reporter)

Updated

4 months ago
tracking-fennec: --- → ?
status-firefox56: --- → affected
status-firefox57: --- → affected
[triage@0816] Sebastian, is this also AS related, like 1389710? Feel free to remove the [MobileAS] if I am wrong. Thanks.
Flags: needinfo?(s.kaspari)
Whiteboard: [MobileAS]
(Reporter)

Comment 2

4 months ago
I don't see how this might be AS related, but I haven't looked into possible causes either.

Switching browser language to Russian, I see that list grow to four items, Yandex being added.
(Reporter)

Comment 3

4 months ago
I'm learning more about this. Here are per-locale search configurations: https://transvision.mozfr.org/productization/?locale=en-ZA&product=mobile

My default system locale (English Canada) seems to have a list that's the same as under en-ZA (south africa?). I don't see en-CA locale as an option in the URL above; perhaps something changed around handling of locales not explicitly configured?

Also, as a sidenote, in https://bugzilla.mozilla.org/show_bug.cgi?id=1390722#c3 flod says that at some point DuckDuckGo was explicitly excluded from some of the locales.
From triage: this isn't AS related, as far as we know. Please retriage.
Flags: needinfo?(s.kaspari)
Whiteboard: [MobileAS]

Comment 5

4 months ago
Hi Joe, Wesly
Please help me prioritize this. This looks like something we should fix asap.
Thanks!
tracking-fennec: ? → +
Flags: needinfo?(wehuang)
Flags: needinfo?(jcheng)
Keywords: regression
let's decide during sprint planning
Flags: needinfo?(jcheng)
suggest to prioritize it in 57.3, I added to the planning notes
Flags: needinfo?(wehuang)
Whiteboard: [FNC][SPT57.3][MVP]
Because we need to fix this for 56 as well, could you move the priority to fix this as soon as possible? We need time to do the uplift to beta, and I'd like that not to be at the last minute. Thanks.
Flags: needinfo?(wehuang)
Marking this as a 56 blocker. We shouldn't release 56 with the search engine options missing
tracking-firefox56: --- → blocking
No problem, Liz. This is already planned in the current sprint, and I just talked to the team to put it as the 1st priority to work on after completing the task in hand. (i.e. to investigate this one later today or tomorrow)

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #8)
> Because we need to fix this for 56 as well, could you move the priority to
> fix this as soon as possible? We need time to do the uplift to beta, and I'd
> like that not to be at the last minute. Thanks.
Flags: needinfo?(wehuang)

Comment 11

4 months ago
Below is the flow how Fennec gets visible search engines:

1. `SearchPreferenceCategory` dispatches a message "SearchEngines:GetVisible" to gecko and expects to receive an array of search engines through message "SearchEngines:Data" sent from browser.js
https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/preferences/SearchPreferenceCategory.java#43,76

2. browser.js get the search engine array by calling `Services.search.getVisibleEngines()`.
https://dxr.mozilla.org/mozilla-central/source/mobile/android/chrome/content/browser.js#6132,6162

And it looks like the logic to compose a search engine array is came from nsSearchService.js.
https://dxr.mozilla.org/mozilla-central/source/toolkit/components/search/nsSearchService.js#3909

It's probably platform's work. :snorp would you mind take a look?
Flags: needinfo?(snorp)
Looks like Mike Kaply has touched this most recently
Flags: needinfo?(snorp) → needinfo?(mozilla)

Comment 13

4 months ago
I can recreate. Looking.
Flags: needinfo?(mozilla)

Comment 14

4 months ago
So this is actually a core Fennec bug. en-CA is being mapped to en-ZA internally.

You can see this by changing your system locale to en-CA and then deleting all data and restarting Fennec.

Then go to:

chrome://global/locale/intl.properties

You'll see the properties file for en-ZA.

My guy says this is related to bug 1378501 where aren't properly doing locale switching, but I can't be sure.

This shouldn't be a new issue - even before, we would have been using the list.txt file for en-ZA.
See Also: → bug 1378501

Comment 15

3 months ago
I had done some study last weekend but still have no idea what the root cause is. Since comment 0 said this is a regression in 56, let's ask for regression window to find the culprit.

Hi Ioana, would you mind help us find the regression window?
Flags: needinfo?(ioana.chiorean)
Keywords: regressionwindow-wanted

Comment 16

3 months ago
Yes, Jing-wei Wu - Mihai from our team will work on it.
Flags: needinfo?(ioana.chiorean) → needinfo?(mihai.ninu)

Comment 17

3 months ago
Hi everyone, 

I don't think this is a 56 regression since I could track it back to and reproduce it also on 55.0a1 (2017-06-02) build, with the same reproduction steps mentioned by Mike @ comm 14.
Flags: needinfo?(mihai.ninu)

Comment 18

3 months ago
(In reply to Mihai Ninu {:Ninu} from comment #17)
> Hi everyone, 
> 
> I don't think this is a 56 regression since I could track it back to and
> reproduce it also on 55.0a1 (2017-06-02) build, with the same reproduction
> steps mentioned by Mike @ comm 14.

This is weird. If the search engines are identical in 55/56/57, :grisha shouldn't find the difference. 

Perhaps different devices/version causes different result? :grisha could you help provide your environment(device model, version, etc.) and reproduced steps?
Flags: needinfo?(gkruglov)
(Reporter)

Comment 19

3 months ago
I've just tried this now on latest 55, 56 and 57. When all are set to "default system language", en-CA, all versions show the same list (see Comment 1). Mike's assessment that it's not a new regression seems on point.

Perhaps updating (Comment 1) reset my language settings in the browser, and that's why I perceived the list to have changed? I recall filing locale switching bugs around that time, so it's possible I've had the locale configured to something other than default en-CA, perhaps en-US, which would display a larger list of search engines. If so, that's a separate bug we should test for.
Flags: needinfo?(gkruglov)
(Reporter)

Updated

3 months ago
Summary: Only three default search engines on 56 Beta and 57 Nightly → Locales are wrongly mapped in Fennec, causing an incorrect list of search engines to be displayed
(Reporter)

Updated

3 months ago
status-firefox55: --- → affected
(Reporter)

Comment 20

3 months ago
(In reply to Mike Kaply [:mkaply] from comment #14)
> So this is actually a core Fennec bug. en-CA is being mapped to en-ZA
> internally.
I'm not familiar with this part of Fennec. What's the real scope of this bug then? What else can this be affecting?
Flags: needinfo?(mozilla)

Comment 21

3 months ago
> I'm not familiar with this part of Fennec. What's the real scope of this bug then? What else can this be affecting?

It could be mapping any cases where the user selects a locale in Android that doesn't map 1:1 to an official locale in Firefox.

so somehow we're choosing en-ZA for en-CA instead of en-US. There defintieyl could be others.
Flags: needinfo?(mozilla)
(Reporter)

Updated

3 months ago
Blocks: 1398209
(Assignee)

Comment 22

3 months ago
> so somehow we're choosing en-ZA for en-CA instead of en-US. There defintieyl could be others.

In Firefox Desktop we switched to use LocaleService::NegotiateLanguages API [0] which is a C++ implementation of the fluent-langneg algorithm [1].

The algorithm is the best approach to language negotiation we were able to produce and it handles such cases really well. Since Fennec is treating LocaleService as a client, and does negotiation on it's own, we only use it in Fennec to match the Gecko locales to whatever Fennec's Java code sends us.

I would recommend Fennec to use the same algorithm to negotiate locales in the Java process, either by calling to Gecko's API, porting the code to Java, or using the JavaScript version. :)


[0] http://searchfox.org/mozilla-central/source/intl/locale/LocaleService.h#239
[1] https://github.com/projectfluent/fluent.js/tree/master/fluent-langneg
(In reply to Mike Kaply [:mkaply] from comment #14)
> So this is actually a core Fennec bug. en-CA is being mapped to en-ZA
> internally.

Do you mean "internally, by Gecko"?
Flags: needinfo?(mozilla)

Comment 24

3 months ago
> Do you mean "internally, by Gecko"?

Yes, I believe Gecko internally is mapping en-CA to en-ZA somehow.

I think we'll have a better answer after the other locale fixes.
Flags: needinfo?(mozilla)
Note, the fixes we are looking to test landed in bug 1378501 and bug 1397925.
My suspicion, btw, is that those fixes will _not_ be related to this bug -- they cause Gecko to correctly track the OS locale, and the issue here seems to be that Gecko _is_ correctly tracking the OS locale, but producing an undesirable fallback.

If we're really falling back to South African English for Canadians, and we're doing so for strings and for search engines, then :horrified face:.
(Assignee)

Comment 27

3 months ago
I tried to reproduce it today, but unfortunately the Galaxy S4 I received for my Fennec debugging does not have English (Canada) in OS locales. I tried setting it to English (Australia), and the chrome://global/locale/intl.properties correctly points to en-US.

Is there any other combination of locales that I could test against? I'm not sure how to get en-CA on Android.

Comment 28

3 months ago
This is NOT fixed by bug 1378501 and bug 1397925.

If you can point me where in the code I can start debugging this, I'll take a look.
(Assignee)

Comment 29

3 months ago
Ok, for starters, look at the three prefs:

1) intl.locale.matchOS
2) general.useragent.locale
3) intl.locale.os

(1) should be true/false depending on if you follow OS locales or not. (2) should have the value of the UI locale selected in Fennec settings (matters only if you don't follow OS), (3) should have the value of the currently selected UI locale in Android.

Next:

1) OSPrefs:

const osPrefs = Cc["@mozilla.org/intl/ospreferences;1"].
  getService(Ci.mozIOSPreferences);
osPrefs.getSystemLocales();

should have the same value as `intl.locale.os`

2) LocaleService:

Services.locale.getAppLocalesAsLangTags();
Services.locale.getAvailableLocales();
Services.locale.getRequestedLocales();

I should be able to tell you much more when I see the values you get out of those APIs.
Mike, are you going to take this bug?
status-firefox55: affected → wontfix
Flags: needinfo?(mozilla)
Snorp, Joe, is there anyone you can assign this to? It's currently blocking 56.
Flags: needinfo?(snorp)
Flags: needinfo?(jcheng)

Comment 32

3 months ago
I'm planning to do some debugging tonight. I'm in a class today. I don't want to take per say in case someone else has cycles, but it will be my priority this evening.
Flags: needinfo?(mozilla)
I'll find time to take a look during the day, and keep this bug updated so I can hand off to mkaply.
I did a build with en-ZA, fr, en-US. I set my OS locale to en-GB (no en-CA on this device).

Fennec is happy:

09-13 12:03:22.715 D/GeckoApp(22452): OS locale is en_GB, app locale is null

Fennec's search engine manager tries `en-GB`, then `en`, then falls back to `en-US`, grabbing the list correctly:

09-13 12:03:22.860 D/GeckoSearchEngineManager(22452): Found default engine name in SharedPreferences: Google
09-13 12:03:22.860 D/GeckoSearchEngineManager(22452): cEFL: Google
09-13 12:03:22.860 D/GeckoSearchEngineManager(22452): getSearchPluginsJarURL: chrome/en-GB/locale/en-GB/browser/searchplugins/list.json
09-13 12:03:22.870 D/GeckoJarReader(22452): No Entry for chrome/en-GB/locale/en-GB/browser/searchplugins/list.json
09-13 12:03:22.870 D/GeckoSearchEngineManager(22452): getSearchPluginsJarURL: chrome/en/locale/en/browser/searchplugins/list.json
09-13 12:03:22.882 D/GeckoJarReader(22452): No Entry for chrome/en/locale/en/browser/searchplugins/list.json
09-13 12:03:22.882 D/GeckoSearchEngineManager(22452): Using fallback search plugin.
09-13 12:03:22.883 D/GeckoSearchEngineManager(22452): Fallback locale: en-US
09-13 12:03:22.883 D/GeckoSearchEngineManager(22452): getSearchPluginsJarURL: chrome/en-US/locale/en-US/browser/searchplugins/list.json

And it recognizes that I'm in the US:

09-13 12:03:22.885 D/GeckoSearchEngineManager(22452): cEFL: region is US

And it looks up each locale in the correct locale, failing each time in the same way back to US.

09-13 12:03:22.885 D/GeckoSearchEngineManager(22452): getSearchPluginsJarURL: chrome/en-GB/locale/en-GB/browser/searchplugins/yahoo.xml
09-13 12:03:22.892 V/GeckoDLCCatalog(22452): Waiting for catalog to be loaded
09-13 12:03:22.901 D/GeckoDLCCatalog(22452): Loading from disk
09-13 12:03:22.903 D/GeckoJarReader(22452): No Entry for chrome/en-GB/locale/en-GB/browser/searchplugins/yahoo.xml
09-13 12:03:22.903 D/GeckoSearchEngineManager(22452): getSearchPluginsJarURL: chrome/en/locale/en/browser/searchplugins/yahoo.xml
09-13 12:03:22.903 D/GeckoJarReader(22452): No Entry for chrome/en/locale/en/browser/searchplugins/yahoo.xml
09-13 12:03:22.904 D/GeckoSearchEngineManager(22452): Using fallback search plugin.
09-13 12:03:22.904 D/GeckoSearchEngineManager(22452): Fallback locale: en-US
09-13 12:03:22.904 D/GeckoSearchEngineManager(22452): getSearchPluginsJarURL: chrome/en-US/locale/en-US/browser/searchplugins/yahoo.xml
09-13 12:03:22.905 D/GeckoSearchEngineManager(22452): getSearchPluginsJarURL: chrome/en-GB/locale/en-GB/browser/searchplugins/google-nocodes.xml
09-13 12:03:22.917 D/GeckoDLCCatalog(22452): Loaded 48 elements
09-13 12:03:22.922 D/GeckoJarReader(22452): No Entry for chrome/en-GB/locale/en-GB/browser/searchplugins/google-nocodes.xml
09-13 12:03:22.922 D/GeckoSearchEngineManager(22452): getSearchPluginsJarURL: chrome/en/locale/en/browser/searchplugins/google-nocodes.xml
09-13 12:03:22.923 D/GeckoJarReader(22452): No Entry for chrome/en/locale/en/browser/searchplugins/google-nocodes.xml
09-13 12:03:22.923 D/GeckoSearchEngineManager(22452): Using fallback search plugin.
09-13 12:03:22.923 D/GeckoSearchEngineManager(22452): Fallback locale: en-US
09-13 12:03:22.923 D/GeckoSearchEngineManager(22452): getSearchPluginsJarURL: chrome/en-US/locale/en-US/browser/searchplugins/google-nocodes.xml
09-13 12:03:22.924 D/GeckoSearchEngineManager(22452): cEFL: returning org.mozilla.gecko.search.SearchEngine@84582e



getAvailableLocales on the Gecko side is quietly logging the available and requested locales correctly:

09-13 12:03:23.044 I/Gecko   (22452): Available locale: en-ZA
09-13 12:03:23.044 I/Gecko   (22452): Available locale: fr
09-13 12:03:23.044 I/Gecko   (22452): Available locale: en-US
09-13 12:03:23.044 I/Gecko   (22452): Available locale: en-ZA
09-13 12:03:23.044 I/Gecko   (22452): Available locale: fr
09-13 12:03:23.044 I/Gecko   (22452): Available locale: en-US

09-13 12:03:23.579 I/Gecko   (22452): Requested locales: en-GB
When I tap in the URL bar and start typing, I get Google, Twitter, Wikipedia:

09-13 12:11:29.858 D/GeckoSearchEngine(22452): Search engine: google-nocodes, Google
09-13 12:11:29.861 D/GeckoSearchEngine(22452): Search engine: twitter, Twitter
09-13 12:11:29.862 D/GeckoSearchEngine(22452): Search engine: wikipedia, Wikipedia
If I forcibly swizzle en-CA in GeckoApplication.onCreate, we appear to fall back correctly:

09-13 12:29:13.361 D/GeckoSearchEngineManager(23343): Found region-specific default engine name in browsersearch.json.
09-13 12:29:13.363 D/GeckoSearchEngineManager(23343): cEFL: Yahoo
09-13 12:29:13.364 D/GeckoSearchEngineManager(23343): getSearchPluginsJarURL: chrome/en-CA/locale/en-CA/browser/searchplugins/list.json
09-13 12:29:15.992 W/GeckoLinker(23343): /data/app/org.mozilla.fennec_rnewman-1/base.apk!/assets/armeabi-v7a/libxul.so: unhandled flags #8 not handled
09-13 12:29:15.997 W/GeckoLinker(23343): /data/app/org.mozilla.fennec_rnewman-1/base.apk!/assets/armeabi-v7a/liblgpllibs.so: unhandled flags #8 not handled
09-13 12:29:16.020 E/GeckoLibLoad(23343): Loaded libs in 4059.035165ms total, 2660ms(3710ms) user, 220ms(460ms) system, 2(17) faults
09-13 12:29:16.023 D/GeckoJarReader(23343): No Entry for chrome/en-CA/locale/en-CA/browser/searchplugins/list.json
09-13 12:29:16.023 D/GeckoSearchEngineManager(23343): getSearchPluginsJarURL: chrome/en/locale/en/browser/searchplugins/list.json
09-13 12:29:16.023 D/GeckoJarReader(23343): No Entry for chrome/en/locale/en/browser/searchplugins/list.json
09-13 12:29:16.023 D/GeckoSearchEngineManager(23343): Using fallback search plugin.
09-13 12:29:16.024 W/GeckoThread(23343): zerdatime 505468119 - runGecko
09-13 12:29:16.024 D/GeckoSearchEngineManager(23343): Fallback locale: en-US
09-13 12:29:16.024 D/GeckoSearchEngineManager(23343): getSearchPluginsJarURL: chrome/en-US/locale/en-US/browser/searchplugins/list.json
09-13 12:29:16.025 D/GeckoSearchEngineManager(23343): cEFL: region is US
09-13 12:29:16.025 D/GeckoSearchEngineManager(23343): getSearchPluginsJarURL: chrome/en-CA/locale/en-CA/browser/searchplugins/yahoo.xml
09-13 12:29:16.026 D/GeckoJarReader(23343): No Entry for chrome/en-CA/locale/en-CA/browser/searchplugins/yahoo.xml
09-13 12:29:16.026 D/GeckoSearchEngineManager(23343): getSearchPluginsJarURL: chrome/en/locale/en/browser/searchplugins/yahoo.xml
09-13 12:29:16.026 D/GeckoJarReader(23343): No Entry for chrome/en/locale/en/browser/searchplugins/yahoo.xml
09-13 12:29:16.026 D/GeckoSearchEngineManager(23343): Using fallback search plugin.
09-13 12:29:16.026 D/GeckoSearchEngineManager(23343): Fallback locale: en-US
09-13 12:29:16.026 D/GeckoSearchEngineManager(23343): getSearchPluginsJarURL: chrome/en-US/locale/en-US/browser/searchplugins/yahoo.xml
09-13 12:29:16.029 D/GeckoSearchEngineManager(23343): cEFL: returning org.mozilla.gecko.search.SearchEngine@2cf7145d

Comment 37

3 months ago
What do you see when you navigate to

chrome://global/locale/intl.properties?
If I fake out the region, too (to the string "CA"), I get expected behavior, as far as I can tell: we oddly list Google as default in browsersearch.json, but we load the US searchplugins/list.json, and end up with Yahoo as default, then Google, Bing, Amazon, DDG, Twitter, and Wikipedia.

09-13 12:47:05.559 D/GeckoSearchEngineManager(24384): Found default engine name in browsersearch.json: Google
09-13 12:47:05.559 D/GeckoSearchEngineManager(24384): cEFL: Google
09-13 12:47:05.559 D/GeckoSearchEngineManager(24384): getSearchPluginsJarURL: chrome/en-CA/locale/en-CA/browser/searchplugins/list.json

09-13 12:47:09.004 D/GeckoJarReader(24384): No Entry for chrome/en-CA/locale/en-CA/browser/searchplugins/list.json
09-13 12:47:09.005 D/GeckoSearchEngineManager(24384): getSearchPluginsJarURL: chrome/en/locale/en/browser/searchplugins/list.json
09-13 12:47:09.006 D/GeckoJarReader(24384): No Entry for chrome/en/locale/en/browser/searchplugins/list.json
09-13 12:47:09.006 D/GeckoSearchEngineManager(24384): Using fallback search plugin.

09-13 12:47:09.009 D/GeckoSearchEngineManager(24384): Fallback locale: en-US
09-13 12:47:09.009 D/GeckoSearchEngineManager(24384): getSearchPluginsJarURL: chrome/en-US/locale/en-US/browser/searchplugins/list.json
09-13 12:47:09.020 D/GeckoSearchEngineManager(24384): cEFL: region is CA
09-13 12:47:09.021 D/GeckoSearchEngineManager(24384): getSearchPluginsJarURL: chrome/en-CA/locale/en-CA/browser/searchplugins/google-nocodes.xml
09-13 12:47:09.021 D/GeckoJarReader(24384): No Entry for chrome/en-CA/locale/en-CA/browser/searchplugins/google-nocodes.xml
09-13 12:47:09.021 D/GeckoSearchEngineManager(24384): getSearchPluginsJarURL: chrome/en/locale/en/browser/searchplugins/google-nocodes.xml
09-13 12:47:09.022 D/GeckoJarReader(24384): No Entry for chrome/en/locale/en/browser/searchplugins/google-nocodes.xml
09-13 12:47:09.022 D/GeckoSearchEngineManager(24384): Using fallback search plugin.
09-13 12:47:09.022 D/GeckoSearchEngineManager(24384): Fallback locale: en-US
09-13 12:47:09.022 D/GeckoSearchEngineManager(24384): getSearchPluginsJarURL: chrome/en-US/locale/en-US/browser/searchplugins/google-nocodes.xml
09-13 12:47:09.027 D/GeckoSearchEngineManager(24384): cEFL: returning org.mozilla.gecko.search.SearchEngine@2cf7145d

09-13 12:47:09.076 I/Gecko   (24384): Available locales are empty.
09-13 12:47:09.076 I/Gecko   (24384): Available locale: en-ZA
09-13 12:47:09.076 I/Gecko   (24384): Available locale: fr
09-13 12:47:09.076 I/Gecko   (24384): Available locale: en-US

09-13 12:47:18.855 D/GeckoSearchEngine(24384): Search engine: yahoo, Yahoo
09-13 12:47:18.856 D/GeckoSearchEngine(24384): Search engine: google-nocodes, Google
09-13 12:47:18.862 D/GeckoSearchEngine(24384): Search engine: bing, Bing
09-13 12:47:18.863 D/GeckoSearchEngine(24384): Search engine: amazondotcom, Amazon.com
09-13 12:47:18.874 D/GeckoSearchEngine(24384): Search engine: duckduckgo, DuckDuckGo
09-13 12:47:18.874 D/GeckoSearchEngine(24384): Search engine: twitter, Twitter
09-13 12:47:18.875 D/GeckoSearchEngine(24384): Search engine: wikipedia, Wikipedia



In this setup, if I load chrome://…/intl.properties, I get:

g.u.l = en-US
accept_lang = en-US, en
lang group = x-western
This might be a great time to note that the Fennec frontend doesn't use Gecko for search engine handling; it looks in the jar directly via `SearchEngineManager.java`.

I don't know why Settings messages Gecko; that seems like it would be ripe for bugs, though on my test device the list is the same.

Grisha, on a clean en-CA device, do you see the same set of search engines when you tap the URL bar and start typing as you see in Settings?
Flags: needinfo?(gkruglov)
(Reporter)

Comment 40

3 months ago
I see the same thing in the URL bar UI and in Settings. On a clean, en-CA device I see Google (default), Twitter, Wikipedia.

First launch:

09-13 13:29:40.967 11019 11077 D GeckoApp: OS locale is en_CA, app locale is null
09-13 13:29:41.321 11019 11019 D GeckoBrowserApp: onLocaleReady: en_CA
09-13 13:29:41.770 11019 11077 D GeckoSearchEngineManager: Found default engine name in browsersearch.json.

09-13 13:29:46.595 11019 11077 D GeckoJarReader: No Entry for chrome/en-CA/locale/en-CA/browser/searchplugins/list.json
09-13 13:29:46.596 11019 11077 D GeckoJarReader: No Entry for chrome/en/locale/en/browser/searchplugins/list.json
09-13 13:29:46.603 11019 11077 D GeckoJarReader: No Entry for chrome/en-CA/locale/en-CA/browser/searchplugins/google-nocodes.xml
09-13 13:29:46.604 11019 11077 D GeckoJarReader: No Entry for chrome/en/locale/en/browser/searchplugins/google-nocodes.xml
09-13 13:29:46.624 11019 11079 I GeckoConsole: Could not read chrome manifest 'file:///data/user/0/org.mozilla.fennec_aurora/chrome.manifest'.
09-13 13:29:46.644 11019 11077 D GeckoApplication: Running post-distribution task: android preferences.

en-ZA pops up:

09-13 13:29:48.466 11019 11079 I GeckoConsole: Locale:OS: en-CA
09-13 13:29:48.476 11019 11079 I GeckoConsole: New OS locale.
09-13 13:29:48.496 11019 11079 I GeckoConsole: Default intl.accept_languages = en-ZA, en-GB, en-US, en
09-13 13:29:48.508 11019 11079 I GeckoConsole: Setting intl.accept_languages to en-ca,en-za,en-gb,en-us,en

All other java-side logging looks normal:

09-13 13:30:32.823 11019 11019 V GeckoPreferences: Checking locale: en_CA vs en_CA
09-13 13:30:35.668 11019 11019 V GeckoPreferences: Checking locale: en_CA vs en_CA
09-13 13:30:39.994 11019 11019 V GeckoPreferences: Checking locale: en_CA vs en_CA
09-13 13:30:42.124 11019 11019 V GeckoPreferences: Checking locale: en_CA vs en_CA
09-13 13:30:43.683 11019 11019 V GeckoPreferences: Checking locale: en_CA vs en_CA
09-13 13:30:43.711 11019 11019 D GeckoLocaleList: Building locales list. Current locale: en_CA
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Aragonese => an
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Asturian => ast
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Azərbaycan dili => az
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Bahasa Malaysia => ms
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Brezhoneg => br
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Cak => cak
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Català => ca
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Čeština => cs
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Cymraeg => cy
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Dansk => da
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Deutsch => de
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Dolnoserbšćina => dsb
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Eesti => et
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: English (South Africa) => en-ZA
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: English (United Kingdom) => en-GB
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: English (United States) => en-US
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Español (Argentina) => es-AR
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Español (Chile) => es-CL
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Español (España) => es-ES
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Español (México) => es-MX
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Esperanto => eo
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Euskara => eu
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Français => fr
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Gaeilge (Éire) => ga-IE
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Gàidhlig => gd
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Galego => gl
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Guarani => gn
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Hornjoserbšćina => hsb
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Hrvatski => hr
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Indonesia => id
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Íslenska => is
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Italiano => it
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Latviešu => lv
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Lietuvių => lt
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Magyar => hu
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Maithili => mai
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Nederlands => nl
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Norsk bokmål (Norge) => nb-NO
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Nynorsk (Noreg) => nn-NO
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: O‘zbek => uz
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Polski => pl
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Português (Brasil) => pt-BR
09-13 13:30:44.217 11019 11019 V GeckoLocaleList: Português (Portugal) => pt-PT
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Pulaar => ff
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Română => ro
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Rumantsch => rm
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Shqip => sq
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Slovenčina => sk
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Slovenščina => sl
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Son => son
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Suomi => fi
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Svenska (Sverige) => sv-SE
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Taqbaylit => kab
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Türkçe => tr
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: West-Frysk (Nederlân) => fy-NL
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Wolof => wo
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Xhosa => xh
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Zam => zam
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Ελληνικά => el
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Беларуская => be
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Български => bg
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Қазақ тілі => kk
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Русский => ru
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Српски => sr
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Українська => uk
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: ქართული => ka
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: Հայերեն (Հայաստան) => hy-AM
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: עברית => he
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: اردو => ur
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: العربية => ar
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: فارسی => fa
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: मराठी => mr
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: हिन्दी (भारत) => hi-IN
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: অসমীয়া => as
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: বাংলা (ভারত) => bn-IN
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: ਪੰਜਾਬੀ (ਭਾਰਤ) => pa-IN
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: ગુજરાતી (ભારત) => gu-IN
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: ଓଡ଼ିଆ => or
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: தமிழ் => ta
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: తెలుగు => te
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: ಕನ್ನಡ => kn
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: മലയാളം => ml
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: ไทย => th
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: ລາວ => lo
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: ဗမာ => my
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: 한국어 => ko
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: 中文 (中国) => zh-CN
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: 中文 (台灣) => zh-TW
09-13 13:30:44.218 11019 11019 V GeckoLocaleList: 日本語 => ja

09-13 13:30:45.155 11019 11019 V GeckoPreferences: Checking locale: en_CA vs en_CA
09-13 13:30:45.728 11019 11019 V GeckoPreferences: Checking locale: en_CA vs en_CA
09-13 13:30:46.373 11019 11019 D GeckoBrowserApp: No locale change returning from preferences; nothing to do.


Subsequent launch:

09-13 13:58:34.432 27064 27078 D GeckoApp: OS locale is en_CA, app locale is null
09-13 13:58:34.465 27064 27078 D GeckoSearchEngineManager: Found default engine name in SharedPreferences: Google
09-13 13:58:34.468 27064 27078 D GeckoJarReader: No Entry for chrome/en-CA/locale/en-CA/browser/searchplugins/list.json
09-13 13:58:34.469 27064 27078 D GeckoJarReader: No Entry for chrome/en/locale/en/browser/searchplugins/list.json
09-13 13:58:34.474 27064 27078 D GeckoJarReader: No Entry for chrome/en-CA/locale/en-CA/browser/searchplugins/google-nocodes.xml
09-13 13:58:34.474 27064 27078 D GeckoJarReader: No Entry for chrome/en/locale/en/browser/searchplugins/google-nocodes.xml
09-13 13:58:34.567 27064 27064 D GeckoBrowserApp: onLocaleReady: en_CA


Going to add some extra logging and report back.
Flags: needinfo?(gkruglov)
(Reporter)

Comment 41

3 months ago
> 09-13 13:29:48.466 11019 11079 I GeckoConsole: Locale:OS: en-CA
> 09-13 13:29:48.476 11019 11079 I GeckoConsole: New OS locale.
> 09-13 13:29:48.496 11019 11079 I GeckoConsole: Default intl.accept_languages = en-ZA, en-GB, en-US, en
> 09-13 13:29:48.508 11019 11079 I GeckoConsole: Setting intl.accept_languages to en-ca,en-za,en-gb,en-us,en

Do we end up discarding en-ca, and picking the first thing from the intl.accept_languages list, which happens to be en-ZA?

/me starts grepping for how intl.accept_languages is used
Your logging suggests that the frontend knows the OS locale correctly, but Gecko for some reason has picked en-ZA, evidenced by its presence at the front of the default intl.accept_languages.
(Assignee)

Comment 43

3 months ago
Gecko is "picking up" based on negotiateLanguages API.

So, if you can check
1) Services.locale.getRequestedLocales()
2) Services.locale.getAvailableLocales();

and the requested shows 'en-CA', and the available shows 'en-US' and 'en-ZA' and then

3) Services.locale.getAppLocalesAsLangTags() shows 'en-ZA' that would mean that our algorithm is making a mistake.
(Reporter)

Comment 44

3 months ago
LocaleService.getRequestedLocales()
Array [ "en-CA" ]

LocaleService.getAvailableLocales()
0 : "zh-TW"
1 : "zh-CN"
2 : "zam"
3 : "xh"
4 : "wo"
5 : "uz"
6 : "ur"
7 : "uk"
8 : "tr"
9 : "th"
10 : "te"
11 : "ta"
12 : "sv-SE"
13 : "sr"
14 : "sq"
15 : "son"
16 : "sl"
17 : "sk"
18 : "ru"
19 : "ro"
20 : "rm"
21 : "pt-PT"
22 : "pt-BR"
23 : "pl"
24 : "pa-IN"
25 : "or"
26 : "nn-NO"
27 : "nl"
28 : "nb-NO"
29 : "my"
30 : "ms"
31 : "mr"
32 : "ml"
33 : "mai"
34 : "lv"
35 : "lt"
36 : "lo"
37 : "ko"
38 : "kn"
39 : "kk"
40 : "kab"
41 : "ka"
42 : "ja"
43 : "it"
44 : "is"
45 : "id"
46 : "hy-AM"
47 : "hu"
48 : "hsb"
49 : "hr"
50 : "hi-IN"
51 : "he"
52 : "gu-IN"
53 : "gn"
54 : "gl"
55 : "gd"
56 : "ga-IE"
57 : "fy-NL"
58 : "fr"
59 : "fi"
60 : "ff"
61 : "fa"
62 : "eu"
63 : "et"
64 : "es-MX"
65 : "es-ES"
66 : "es-CL"
67 : "es-AR"
68 : "eo"
69 : "en-ZA"
70 : "en-GB"
71 : "el"
72 : "dsb"
73 : "de"
74 : "da"
75 : "cy"
76 : "cs"
77 : "cak"
78 : "ca"
79 : "br"
80 : "bn-IN"
81 : "bg"
82 : "be"
83 : "az"
84 : "ast"
85 : "as"
86 : "ar"
87 : "an"
88 : "en-US"

LocaleService.getAppLocalesAsLangTags()
Array [ "en-ZA", "en-GB", "en-US" ]
(Assignee)

Comment 45

3 months ago
Reproduced!

Services.locale.negotiateLanguages(['en-CA'], ['en-ZA','en-US'], 'en-US') === ['en-ZA', 'en-US']


It's also probably the same bug as bug 1399203.

Investigating.
(Reporter)

Comment 46

3 months ago
Custom multi-locale (fr, en-ZA, en-US) build with some extra logging seems to pick a correct fallback, en-US. 

09-13 15:15:33.805  6112  6131 D GeckoApp: OS locale is en_CA, app locale is null
09-13 15:15:34.044  6112  6112 D GeckoBrowserApp: onLocaleReady: en_CA
09-13 15:15:34.559  6112  6131 D GeckoSearchEngineManag: Found default engine name in browsersearch.json: Google
09-13 15:15:34.559  6112  6131 D GeckoSearchEngineManag: cEFL: Google
09-13 15:15:34.559  6112  6131 D GeckoSearchEngineManag: getSearchPluginsJarURL: chrome/en-CA/locale/en-CA/browser/searchplugins/list.json
09-13 15:15:37.435  6112  6131 D GeckoJarReader: No Entry for chrome/en-CA/locale/en-CA/browser/searchplugins/list.json
09-13 15:15:37.436  6112  6131 D GeckoJarReader: No Entry for chrome/en/locale/en/browser/searchplugins/list.json
09-13 15:15:37.439  6112  6131 D GeckoSearchEngineManag: Fallback locale: en-US
09-13 15:15:37.439  6112  6131 D GeckoSearchEngineManag: getSearchPluginsJarURL: chrome/en-US/locale/en-US/browser/searchplugins/list.json
09-13 15:15:37.439  6112  6131 D GeckoSearchEngineManag: Using fallback search plugin. jar:jar:file:/data/app/org.mozilla.fennec_grisha-1/base.apk!/assets/omni.ja!/chrome/en-US/locale/en-US/browser/searchplugins/list.json
09-13 15:15:37.441  6112  6131 D GeckoSearchEngineManag: cEFL: region is CA
09-13 15:15:37.441  6112  6131 D GeckoSearchEngineManag: getSearchPluginsJarURL: chrome/en-CA/locale/en-CA/browser/searchplugins/google-nocodes.xml
09-13 15:15:37.441  6112  6131 D GeckoJarReader: No Entry for chrome/en-CA/locale/en-CA/browser/searchplugins/google-nocodes.xml
09-13 15:15:37.441  6112  6131 D GeckoSearchEngineManag: getSearchPluginsJarURL: chrome/en/locale/en/browser/searchplugins/google-nocodes.xml
09-13 15:15:37.441  6112  6131 D GeckoJarReader: No Entry for chrome/en/locale/en/browser/searchplugins/google-nocodes.xml
09-13 15:15:37.442  6112  6131 D GeckoSearchEngineManag: Fallback locale: en-US
09-13 15:15:37.442  6112  6131 D GeckoSearchEngineManag: getSearchPluginsJarURL: chrome/en-US/locale/en-US/browser/searchplugins/google-nocodes.xml
09-13 15:15:37.442  6112  6131 D GeckoSearchEngineManag: Using fallback search plugin. jar:jar:file:/data/app/org.mozilla.fennec_grisha-1/base.apk!/assets/omni.ja!/chrome/en-US/locale/en-US/browser/searchplugins/google-nocodes.xml
09-13 15:15:37.443  6112  6131 D GeckoSearchEngineManag: cEFL: returning org.mozilla.gecko.search.SearchEngine@67c27eb

09-13 15:15:37.459  6112  6136 I GeckoXXX: Available locales are empty.
09-13 15:15:37.459  6112  6136 I GeckoXXX: Available locales are empty.
09-13 15:15:37.459  6112  6136 I GeckoXXX:   Available locale: en-US
09-13 15:15:37.460  6112  6136 I GeckoXXX: Available locales are empty.
09-13 15:15:37.460  6112  6136 I GeckoXXX:   Available locale: fr
09-13 15:15:37.460  6112  6136 I GeckoXXX:   Available locale: en-US
09-13 15:15:37.460  6112  6136 I GeckoXXX: Available locales are empty.
09-13 15:15:37.460  6112  6136 I GeckoXXX:   Available locale: en-ZA
09-13 15:15:37.460  6112  6136 I GeckoXXX:   Available locale: fr
09-13 15:15:37.460  6112  6136 I GeckoXXX:   Available locale: en-US

09-13 15:15:37.909  6112  6136 I GeckoXXX: Requested locales: en-US

09-13 15:15:38.633  6112  6136 I GeckoXXX: Requested locales: en-US

09-13 15:15:39.125  6112  6136 I GeckoConsole: Locale:OS: en-CA
09-13 15:15:39.126  6112  6136 I GeckoConsole: New OS locale.
09-13 15:15:39.131  6112  6136 I GeckoConsole: Default intl.accept_languages = en-US, en
09-13 15:15:39.133  6112  6136 I GeckoConsole: Setting intl.accept_languages to en-us,en-ca,en

09-13 15:17:38.841  6112  6136 I GeckoXXX: Requested locales: en-US
09-13 15:19:28.233  6112  6112 V GeckoPreferences: Checking locale: en_CA vs en_CA
09-13 15:19:31.140  6112  6112 V GeckoPreferences: Checking locale: en_CA vs en_CA
09-13 15:19:36.035  6112  6112 V GeckoPreferences: Checking locale: en_CA vs en_CA

09-13 15:19:40.579  6112  6112 D GeckoSearchEngine: Search engine: google-nocodes, Google
09-13 15:19:40.581  6112  6112 D GeckoSearchEngine: Search engine: yahoo, Yahoo
09-13 15:19:40.582  6112  6112 D GeckoSearchEngine: Search engine: bing, Bing
09-13 15:19:40.583  6112  6112 D GeckoSearchEngine: Search engine: amazondotcom, Amazon.com
09-13 15:19:40.584  6112  6112 D GeckoSearchEngine: Search engine: duckduckgo, DuckDuckGo
09-13 15:19:40.585  6112  6112 D GeckoSearchEngine: Search engine: twitter, Twitter
09-13 15:19:40.586  6112  6112 D GeckoSearchEngine: Search engine: wikipedia, Wikipedia

09-13 15:19:44.879  6112  6112 V GeckoPreferences: Checking locale: en_CA vs en_CA
09-13 15:19:47.237  6112  6112 V GeckoPreferences: Checking locale: en_CA vs en_CA
09-13 15:19:48.277  6112  6112 V GeckoPreferences: Checking locale: en_CA vs en_CA
09-13 15:19:49.033  6112  6112 V GeckoPreferences: Checking locale: en_CA vs en_CA

09-13 15:19:49.159  6112  6112 D GeckoSearchEngine: Search engine: google-nocodes, Google
09-13 15:19:49.160  6112  6112 D GeckoSearchEngine: Search engine: yahoo, Yahoo
09-13 15:19:49.161  6112  6112 D GeckoSearchEngine: Search engine: bing, Bing
09-13 15:19:49.161  6112  6112 D GeckoSearchEngine: Search engine: amazondotcom, Amazon.com
09-13 15:19:49.162  6112  6112 D GeckoSearchEngine: Search engine: duckduckgo, DuckDuckGo
09-13 15:19:49.162  6112  6112 D GeckoSearchEngine: Search engine: twitter, Twitter
09-13 15:19:49.163  6112  6112 D GeckoSearchEngine: Search engine: wikipedia, Wikipedia
...etc...
(Assignee)

Comment 47

3 months ago
Hmm... it may not be a bug. There's no indication in CLDR that en-US is closer to en-CA than en-ZA or en-GB, so the algorithm just takes them in order.

I'm wondering if there's anything we can reliably do in an internationalization fashion about it (or should we?), or do we just need a dirty-hacked special-case for "if it's not a direct match, always take en-US first if available" which feels wrong and doesn't scale to other locales, but at least solves this one.
(Assignee)

Comment 48

3 months ago
Pike, what do you think?
Flags: needinfo?(l10n)
Can we ship search definitions for a locale without shipping strings? If so, can we just define what we want to happen for en-CA?

If not, because we can't handle more than one locale in the app at once, how hard would it be to do that?

Can we switch Fennec's search settings to use BrowserSearchManager, which solves this bug by decoupling from Gecko?
See Also: → bug 1399203

Comment 50

3 months ago
I don't have a good idea within the 56 timing. Richard's suggestion to do something constructive for search in Canada would work on desktop, I think, but AFAIK, the data we put in builds for desktop and mobile differs.

For language negotiation, likely subtags from cldr would prefer en-US over en-ZA, but that data also comes with challenges that are tricky to evaluate on a day's notice.

I'm also not sure I'm happy with 

Services.locale.negotiateLanguages(['en-CA', 'en-US'], ['en-ZA','en-US'], 'en-US') == [ "en-ZA", "en-US" ]

If that'd return [ "en-US", "en-ZA" ] we could probably hack this up the easiest, but that's also really speculative.
Flags: needinfo?(l10n)
(Assignee)

Comment 51

3 months ago
> likely subtags from cldr would prefer en-US over en-ZA,

Negative. likelySubtags would not affect en-CA, because it's a perfectly complete language tag without anything to specify further.

See: https://github.com/unicode-cldr/cldr-core/blob/master/supplemental/likelySubtags.json

> I'm also not sure I'm happy with 
> Services.locale.negotiateLanguages(['en-CA', 'en-US'], ['en-ZA','en-US'], 'en-US') == [ "en-ZA", "en-US" ]
> If that'd return [ "en-US", "en-ZA" ] we could probably hack this up the easiest, but that's also really speculative.

Understandable.

What I'm saying is that it doesn't seem like our current algorithm sees any reason to prefer en-US over en-ZA for "en-CA". It would of course expand "en" to "en-US" (in fact, to "en-Latn-US"), but if someone asks for "en-CA", we're just looking for an exact match, and if that's missing, we're happy to take all other "en-*" in any order.

The same logic applies to "fr-*" or "de-*" where id the exact match is missing, and the user did specify a region, then we'll try all of them.

One idea would be to add additional step to the logic: if the the exact match is missing and the likely subtag is not available, and the language tag has region specified, remove the region and attempt to get likely subtag again.

This would do:

1) Take `en-CA`
2) Attempt exact match: false
3) AddLikelySubtags: `en-CA`
4) Attempt match: false
5) Strip region: `en`
6) Attempt likelySubtags: `en-US`
7) Attempt match: true

This would work across other pairs like fr-* or de-* picking the most likely region for the locale if the exact region is unavailable.

I can write a patch tomorrow.
(Assignee)

Updated

3 months ago
Blocks: 1400006
(Assignee)

Comment 52

3 months ago
The patch is ready in bug 1400006.
Flags: needinfo?(snorp)
Flags: needinfo?(jcheng)
(Assignee)

Updated

3 months ago
No longer blocks: 1400006
Depends on: 1400006
(Assignee)

Comment 53

3 months ago
This should be fixed in Nightly starting today.

I'm going to close this bug, please reopen if you encounter it again in Nightly.
Status: NEW → RESOLVED
Last Resolved: 3 months ago
Resolution: --- → FIXED
Assignee: nobody → gandalf
status-firefox57: affected → fixed
Target Milestone: --- → Firefox 57
The patch in bug 1400006 is on mozilla-release now so this should be fixed. 
It would be good to verify this issue, mkaply, would you like to do it with a build from https://bugzilla.mozilla.org/show_bug.cgi?id=1400006#c20 ?
status-firefox56: affected → fixed
Flags: needinfo?(mozilla)

Comment 55

3 months ago
Looks great.
Flags: needinfo?(mozilla)
You need to log in before you can comment on or make changes to this bug.