839883 - Don't match zh-* based on just language

Status: RESOLVED FIXED

(Core :: Internationalization, defect)

Description

[Axel Hecht [:Pike]]

We got unwanted behavior in bug 839380 for zh-HK, matching zh-CN.

I think we should just switch off positive matches based on just language for zh-*.

That may or may not violate the spec, which might recover more useful with respecting implied script subtags, but I don't think we need to wait for that.

Comment 1

[Gordon P. Hemsley [:GPHemsley]]

I'm not sure I understand the source of the problem here: There shouldn't be any locale tagged as 'zh', so there shouldn't be any fallback happening from one region-tagged locale to another region-tagged locale. That is to say, 'zh-HK' should not be naively falling back to 'zh-CN' in the first place. AFAIUI, if 'zh-HK' is not available, it should fall back to the native locale (such as 'en-US'), because there should not be any 'zh' to fall back to. (This is the same type of situation that intl.accept_languages is designed to handle, albeit in a different context.)

Comment 2

[Axel Hecht [:Pike]]

Gordon, yes, this is exactly why I filed this bug. We do fall back from zh-HG to zh-TW/zh-CN, and do so randomly, even.

Comment 3

[Gordon P. Hemsley [:GPHemsley]]

(In reply to Axel Hecht [:Pike] from comment #2)
> Gordon, yes, this is exactly why I filed this bug. We do fall back from
> zh-HG to zh-TW/zh-CN, and do so randomly, even.

Oh. Your original summary made it sound like this was expected behavior that you'd like to change, rather than unexpected behavior.

Comment 4

[Mark Finkle (:mfinkle) (use needinfo?)]

Does this have something to do with LanguagesMatch ?
http://mxr.mozilla.org/mozilla-central/source/chrome/src/nsChromeRegistryChrome.cpp#73

Comment 5

[Axel Hecht [:Pike]]

Yes, that'd be the code to change.

Comment 6

[Axel Hecht [:Pike]]

... which also says that this is code that would impact desktop.

Comment 7

[Alex Keybl [:akeybl]]

(In reply to Axel Hecht [:Pike] from comment #6)
> ... which also says that this is code that would impact desktop.

Yeah, we would definitely want a mobile-only change here. Who knows what locales are successfully relying upon this behavior already on the desktop side?

To take a fix here for FF19b6 (instead of pulling out zh-cn for FF19), we need to prepare a patch today.

Comment 8

[henryfhchan]

Sorry I don't know how to code, but currently maybe just special code zh-hk to return false if the other lang is zh-cn, desktop or mobile. 

I have remember seeing bugs where zh-yue font matches zh-cn instead of zh-hk. (Fonts are separately specifiable for Zh-hk for some reason )
so fixing this irrespective desktop or mobile would probably resolve a few other bugs.

In the long term a table should be built for what not to match, but currently one algorithm that will allow singapore and china to intramatch only is:

if ((a=="zh-cn" || a=="zh-sg" || a=="zh-hans") && !(b=="zh-cn" || b=="zh-sg" || b=="zh-hans")) return false;
 if (!(a=="zh-cn" || a=="zh-sg" || a=="zh-hans") && (b=="zh-cn" || b=="zh-sg" || b=="zh-hans")) return false;

(singapore and china use simplified Chinese, taiwan, hong kong and macau traditional.)

Comment 9

[Gordon P. Hemsley [:GPHemsley]]

I think it's better to think about it with what the fallback SHOULD be, rather than what it SHOULDN'T be.

So, maybe something like this:

* zh-Hans, zh-CN, zh-SG(, en-US)
* zh-Hant, zh-TW, zh-HK, zh-MO(, en-US)

(I'm getting a bit of déjà vu, so I've probably said this before, somewhere else.)

Comment 10

[henryfhchan]

Yes, but since this is a function to return a boolean on matching or not (and not returning the fallback path), it makes sense to short circuit and return false for the two possible sets of bad parings, doesn't it....?

Comment 11

[Gordon P. Hemsley [:GPHemsley]]

(In reply to henryfhchan from comment #10)
> Yes, but since this is a function to return a boolean on matching or not
> (and not returning the fallback path), it makes sense to short circuit and
> return false for the two possible sets of bad parings, doesn't it....?

Ah, yes, that might be true. But it'd be a hacky, quick-fix solution just to get it out the door.

Comment 12

[henryfhchan]

We can leave better solutions for Aurora or nightly, a quick and dirty fix is essential for Mozilla not to accidentally step into a pot of boiling water, due to the cultural and political reasons outlined in the bug this is blocking.

Comment 13

[Brad Lassey [:blassey] (use needinfo?)]

Attachment: patch

Comment 14

[(inactive) Jim Chen [:jchen] [:darchons]]

Comment on attachment 712767 [details] [diff] [review]
patch

I'm okay with this as a quick band-aid. Some things to consider for the future,

1) Other locales may have similar issues, and if other locales show up we should consider a more proper fix.

2) Locale.setDefault() doc says the default locale may be overridden when system configuration changes (user changes the system locale). So if the user changes from zh_hk to en_us to zh_hk, the effect of this patch may be overridden.

Comment 15

[Brad Lassey [:blassey] (use needinfo?)]

Comment on attachment 712767 [details] [diff] [review]
patch

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 
User impact if declined: 
Testing completed (on m-c, etc.): 
Risk to taking this patch (and alternatives if risky): 
String or UUID changes made by this patch:

Comment 16

[Alex Keybl [:akeybl]]

Comment on attachment 712767 [details] [diff] [review]
patch

Approving for Aurora/Beta. Thanks all!

Comment 17

[Brad Lassey [:blassey] (use needinfo?)]

Comment on attachment 712767 [details] [diff] [review]
patch

sorry, this patch really belonged on bug 839380, so I moved it over there.

There sounds like there still may be some unexpected behavior in LanguageMatch() that we might want to address in this bug (but can ride the trains)

Comment 18

[Zibi Braniecki [:zbraniecki][:gandalf]]

I think this is fixed now with LocaleService::NegotiateLanguages:

Services.locale.negotiateLanguages(["zh-HK"], ["zh-CN", "zh-TW"]) // Array [ "zh-TW", "zh-CN" ]