839380 - zh-hk locale should use zh-tw translation instead of zh-cn

Status: VERIFIED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[henryfhchan]

a few weeks ago chinese translations landed on nightly.
about a week ago, nightly started using the zh-cn translation for Chinese (Hong Kong) Androids.  This is in error.  The zh-tw translation uses traditional Chinese characters, which is way way more preferred than zh-cn which uses simplified Chinese characters.

The use of simplified vs traditional characters is a part of the conflict in Hong Kong between Hong Kong and Mainland China.  This kind of issue will no doubtly trigger immense emotional response like the Facebook Mobile App did on Android.  It is vital this doesn't ship in Beta or Release.

Comment 1

[Axel Hecht [:Pike]]

Yeah, I can see how this would happen from how language tags are defined. FTR, one reason why we're not supporting zh-HK on the desktop without having this behavior in check on our side.

For Firefox on Android, though, we're relying on Android picking the right language for us.

Seems sad that Android isn't doing a better job here. I don't remember coming across anything that would allow us to hint or tweak that behavior. Brad, Mark, any idea on your side?

Comment 2

[Aaron Train [:aaronmt]]

(In reply to henryfhchan from comment #0)
> a few weeks ago chinese translations landed on nightly.
> about a week ago, nightly started using the zh-cn translation for Chinese
> (Hong Kong) Androids. 

This is written to sound like at one point this was working, or am I reading this wrong?

Comment 3

[Mark Finkle (:mfinkle) (use needinfo?)]

I am completely ignorant of the problem and potential solutions, so let me start dumping what little I know in here:

* https://l10n.mozilla.org/shipping/dashboard shows no zh-hk
* We do seem to have zh-tw and zh-cn
* If a user sets their Android device to "Hong Kong", then Fennec is using zh-cn. That is what this bug is pointing out.

What did I miss or get wrong?

Questions:
* Is Gecko the only part using zh-cn? or is the Java UI using it too?
* Are we falling back to zh-cn because there is a specific mapping to do so, or because we found that locale before zh-tw?
* Are we hoping there is a magic flag to tell Android "if zh-hk is requested, please use zh-tw" ?
* As a fallback, could we copy zh-tw to zh-hk and ship the copied xh-hk in Fennec?

Comment 4

[Axel Hecht [:Pike]]

I don't have answers, trying to CC folks from our l10n communities.

Comment 5

[henryfhchan]

> This is written to sound like at one point this was working, or am I reading
> this wrong?

Yes, at one point it showed the zh-tw translation.

> What did I miss or get wrong?

You are correct.

> * Is Gecko the only part using zh-cn? or is the Java UI using it too?
Only the right-click (ok, long-tap) menu is using zh-cn.  Otherwise, it is in English.

Comment 6

[Peter Pin-Guang Chen [:petercpg] (MozTW.org)]

(In reply to Mark Finkle (:mfinkle) from comment #3)
> I am completely ignorant of the problem and potential solutions, so let me
> start dumping what little I know in here:
> 
> * https://l10n.mozilla.org/shipping/dashboard shows no zh-hk
> * We do seem to have zh-tw and zh-cn
> * If a user sets their Android device to "Hong Kong", then Fennec is using
> zh-cn. That is what this bug is pointing out.
> 
> What did I miss or get wrong?
> 
> Questions:
> * Is Gecko the only part using zh-cn? or is the Java UI using it too?

Like what Henry mentioned, the Java UI displayed in English and Gecko took zh-CN.

> * Are we falling back to zh-cn because there is a specific mapping to do so,
> or because we found that locale before zh-tw?
> * Are we hoping there is a magic flag to tell Android "if zh-hk is
> requested, please use zh-tw" ?
> * As a fallback, could we copy zh-tw to zh-hk and ship the copied xh-hk in
> Fennec?

My device (HTC Butterfly) has only Traditional(zh-TW) and Simplified(zh-CN) Chinese in Android's regional settings. After installing MoreLocale 2 and setting the device locale to zh-HK, only couple of apps (such as Google Maps, Flickr) are displayed in Chinese, others including Chrome, Opera and Android itself are displayed in English. 

Mark's third and fourth point looks good to me. Would there be different impact to HK users?

I would suggest that if we're bringing the same experience from desktop to mobile, we should have zh-HK devices use zh-TW Fennec like what desktop does, or at least use English like what other apps do.

Comment 7

[Irvin (MozTW)]

cc to our newest rep Sammy in HK

Comment 8

[henryfhchan]

both English and Chinese are official languages in Hong Kong, either would suffice.  (Fennec / Fx would be better off with a true zh-hk translation, but that's another bug)

Comment 9

[henryfhchan]

By Chinese I mean Chinese (Traditional).  zh-tw uses Chinese (Traditional) also.

Comment 10

[Sammy Fung]

Hi Henry, can you help how I can re-generate the issue at Firefox on Android ? and which model of Android phone you are using ?

I switched to zh-hk from us-hk and updated my firefox app but long-tap (such as at URL bar) are still showing options in English.

Comment 11

[Sammy Fung]

ok, I heard the case which appears on Firefox beta on Android.

I didn't read the code, so I couldn't confirm on my following opinions. I think probably zh-hk locale is missing in Firefox and fallback to zh which shows in simplified chinese.

The simple lazy solution is copying zh-tw po to zh-hk to avoid above fallback.

Comment 12

[Sammy Fung]

(In reply to Mark Finkle (:mfinkle) from comment #3)
> Questions:
> * Are we hoping there is a magic flag to tell Android "if zh-hk is
> requested, please use zh-tw" ?

In my opinion, it's not suggested to add a special 'magic' flag/trick at Firefox beta. I perfer copying zh-tw po into zh-hk and ship it.

Comment 13

[Axel Hecht [:Pike]]

Ugh, this is worst case in terms of risk. CCing :bs for the chrome registry feedback.

Bottom line:

Android's (java?) language matching algorithm apparently has "just language" match switched off for zh-*, while gecko matches zh-* among each other.

Bug 1: the language selection on the java UI side isn't matching what's happening on the gecko side. Filed bug 839881.
Bug 2: gecko shouldn't match zh-* amongst each other (naivly, at least), filed bug 839883

I'd leave the evaluation of the risk of either approach to those bugs, for each of 19, 20, and 21.

The other alternative would be to pull the feature of adding Chinese.
We could also try to make zh-HK find en-US, either by registring chrome or by really packaging up a zh-HK that just has en-US files, but both smell like breaking release automation to me.

Comment 14

[henryfhchan]

Attachment: about:home

Comment 15

[henryfhchan]

Attachment: long tap on a link

Comment 16

[henryfhchan]

@ sammy 
As you can see it is only apparent on clicking on links, not parts of ui. 
I'm on Samsung Galaxy W, gingerbread. 

I'm not sure about this, but there might be also other zh- androids actually being sold, e.g. zh-sg for Singapore and zh-mo for macau? In my point of view, copying over the translation isn't a good long term solution, instead there should be a way in gecko to specify mappings (e.g. sg, hans to cn; mo, hant, hk to tw)

Comment 17

[henryfhchan]

Anyways, whatever the solution, avoid at all costs shipping Firefox with zh-cn as fallback for zh-hk.  The use of simplified Chinese in Hong Kong is a highly controversial issue.  Facebook came heavily under fire by netizens due to a programming bug that showed zh-cn as fallback for certain strings, and certain Facebook pages even promoted a boycott of the mobile app for android.  Better steer clear of any conflict.

Comment 18

[Alex Keybl [:akeybl]]

What are our options here?

1) Pull out the zh-cn localization entirely, preventing zh-tw from defaulting to zh-cn in some instances
2) Somehow force the longtap menu to fallback to en-US (doesn't seem like we've got ideas yet), and verify that there are no other locations with zh-cn strings on zh-tw
3) Leave things as is (zh-cn mistakenly used in the longtap)

We need to come to a final solution today, as we're going to build with the final FF19 beta.

Comment 19

[Alex Keybl [:akeybl]]

(In reply to Alex Keybl [:akeybl] from comment #18)
> 2) Somehow force the longtap menu to fallback to en-US (doesn't seem like
> we've got ideas yet), and verify that there are no other locations with
> zh-cn strings on zh-tw

Nevermind, I'm reading bug 839883 now. Looks like we're getting closer to a final solution.

Comment 20

[Aaron Train [:aaronmt]]

Attachment: Doorhanger example (Fx Beta)

Just attaching another example screenshot with device set to zh-HK, and loading Firefox Beta, I get this with an example doorhanger.

Comment 21

[Aaron Train [:aaronmt]]

Attachment: Error page example (Fx Beta)

and an in-content error page

Comment 22

[Brad Lassey [:blassey] (use needinfo?)]

Attachment: patch

[Triage Comment]
Patch review in bug 839883, approval for aurora and beta over the phone from akeybl

Comment 23

[Brad Lassey [:blassey] (use needinfo?)]

https://hg.mozilla.org/mozilla-central/rev/36525224b14e
https://hg.mozilla.org/releases/mozilla-aurora/rev/10a9b88b6ac7
https://hg.mozilla.org/releases/mozilla-beta/rev/e5be02aead50

Comment 24

[Aaron Train [:aaronmt]]

With the patch in bug 839883 this seems fixed; testing on trunk. As far as I can tell through comparison; zh-HK is now served a mix of mainly English and zh-TW (traditional characters) through some areas. Henry/Sammy/Peter, can one of you confirm too?

Comment 25

[Peter Pin-Guang Chen [:petercpg] (MozTW.org)]

(In reply to Aaron Train [:aaronmt] from comment #24)
> With the patch in bug 839883 this seems fixed; testing on trunk. As far as I
> can tell through comparison; zh-HK is now served a mix of mainly English and
> zh-TW (traditional characters) through some areas. Henry/Sammy/Peter, can
> one of you confirm too?

I can confirm that now Gecko is displayed in zh-TW, while the Java UI is still in English on latest Nightly when the device is set to zh-HK locale. Not sure if this is desirable, Henry and Sammy?

Comment 26

[henryfhchan]

I would say this is acceptable as a quick fix for shipping without insults.  Obviously it would be great if Firefox would ship with a true zh-hk translation, or unify the locale of the Java UI and Gecko by Bug 839881. :)

Verified fix on the latest nightly.

Comment 27

[henryfhchan]

Accept-Language is still sending zh-cn, is that another bug?

Comment 28

[Axel Hecht [:Pike]]

That's weird, can someone reproduce this?

I'd expect zh-tw, zh, en-us, en

Comment 29

[Aaron Train [:aaronmt]]

Would be a new bug; although I dont see zh-cn, I see zh-tw (using http://web-sniffer.net/; sending e.g, 'mozilla.com') ... Accept-Language: zh-tw,zh;q=0.8;en-us;q=0.5,en;q0.3

Comment 30

[henryfhchan]

@aaronmt For accept-language, please see Bug 841758.

When starting Firefox with tabs automatically restored, right click on any link shows up zh-cn translation. Newly opened tabs also exhibit the erroneous behavior.

However, long tapping in about:home will show up zh-tw correctly. The section header "Top Sites" is also in zh-tw.  The android hardware menu is also in zh-tw.

When starting Firefox without any tabs automatically restored, right click on any link shows up zh-tw translation.  Long-tapping in about:home will also show up zh-tw correctly, however, the header "Top Sites" is in English.  The android hardware menu is still in zh-tw.

Do we have a race condition here? Should this bug be re-opened?

Comment 31

[henryfhchan]

The awesome bar is always in English. The awesome screen is however always in zh-tw.  Both are irrespective of cold start or starting with tabs auto-restoring.  The top right hand corner menu button is also always in English no matter what.

Run on Android 2.3.6, Samsung Galaxy W (with a hardware menu button), zh-hk (Factory)

Comment 32

[henryfhchan]

(In reply to henryfhchan from comment #31)
> The awesome screen is however always  in zh-tw.

I take that back.  The awesome screen is in English if the Gecko backend hasn't started when I tap the awesome bar.  The awesome screen on a new tab is zh-tw after Gecko has started.

Comment 33

[Alex Keybl [:akeybl]]

(In reply to henryfhchan from comment #30)
> When starting Firefox without any tabs automatically restored, right click
> on any link shows up zh-tw translation.  Long-tapping in about:home will
> also show up zh-tw correctly, however, the header "Top Sites" is in English.
> The android hardware menu is still in zh-tw.

Aaron - are you able to repro this?

Comment 34

[Aaron Train [:aaronmt]]

(In reply to henryfhchan from comment #32)
> (In reply to henryfhchan from comment #31)
> > The awesome screen is however always  in zh-tw.
> 
> I take that back.  The awesome screen is in English if the Gecko backend
> hasn't started when I tap the awesome bar.  The awesome screen on a new tab
> is zh-tw after Gecko has started.

The patch added the check in a Gecko init function, it looks like when you are accessing the AwesomeScreen, you are ahead of Gecko availability.

Comment 35

[Aaron Train [:aaronmt]]

@henryfhchan, which build are you seeing hardware menu translations? When I set my device to zh-HK, I only see english

Comment 36

[henryfhchan]

There is a hardware "menu" button on some Android devices, usually in the bottom left or bottom right corner of these handheld phones.

Pressing that button yields the old style menu bar at the bottom.  The menu is consistently in zh-tw.

However the three dots "action overflow" menu are consistently in English, and I think that's the only menu button you will see on a device without the traditional hardware "menu" button.

Comment 37

[Aaron Train [:aaronmt]]

(In reply to henryfhchan from comment #30)
> @aaronmt For accept-language, please see Bug 841758.
> 
> When starting Firefox with tabs automatically restored, right click on any
> link shows up zh-cn translation. Newly opened tabs also exhibit the
> erroneous behavior.

I'm not able to reproduce this with these instructions. What do you mean by automatically restored; clicking the about:home link?

> However, long tapping in about:home will show up zh-tw correctly. The
> section header "Top Sites" is also in zh-tw.  The android hardware menu is
> also in zh-tw.

I see this.

> When starting Firefox without any tabs automatically restored, right click
> on any link shows up zh-tw translation.  Long-tapping in about:home will
> also show up zh-tw correctly, however, the header "Top Sites" is in English.

I only see English if I'm faster than Gecko initialization, easier to reproduce on slower phones. On my Galaxy SII, I dont see the locale change to zh-tw take affect until I back out of the awesome-screen and re-enter. On my Nexus One, I see the locale change to zh-tw take affect after a few seconds.

> The android hardware menu is still in zh-tw.

I see english on my SII (software menu), and zh-tw on my Nexus One.

(In reply to Alex Keybl [:akeybl] from comment #33)
> (In reply to henryfhchan from comment #30)
> > When starting Firefox without any tabs automatically restored, right click
> > on any link shows up zh-tw translation.  Long-tapping in about:home will
> > also show up zh-tw correctly, however, the header "Top Sites" is in English.
> > The android hardware menu is still in zh-tw.
> 
> Aaron - are you able to repro this?

Yes, but fixed by either backing out of the awesome-screen and re-entering (like my SII), on my Nexus One, I just wait a few seconds and when Gecko is init, I see english replaced with zh-tw.

Comment 38

[henryfhchan]

(In reply to Aaron Train [:aaronmt] from comment #37)
> (In reply to henryfhchan from comment #30)
> > @aaronmt For accept-language, please see Bug 841758.
> > 
> > When starting Firefox with tabs automatically restored, right click on any
> > link shows up zh-cn translation. Newly opened tabs also exhibit the
> > erroneous behavior.
> 
> I'm not able to reproduce this with these instructions. What do you mean by
> automatically restored; clicking the about:home link?

Visit a site. Press the home button. Exhaust the memory of the device so android kills Firefox. Start Firefox again. Tabs are restored.

Comment 39

[Aaron Train [:aaronmt]]

Tried to reproduce that 
I/WindowManager( 9379): WIN DEATH: Window{407bf950 org.mozilla.firefox_beta/org.mozilla.firefox_beta.App paused=false}

On re-open it loaded my single tab, I held long-tap on a link, and the context-menu was zh-TW for me (zh-HK locale).

Comment 40

[henryfhchan]

Looks like this is due to https://bugzilla.mozilla.org/show_bug.cgi?id=841758
If the pref is set, gecko will use that language on long tap context menu after automatic restore...

Comment 41

[henryfhchan]

Nvm last comment, I messed up reason with the side effect. 
To reproduce: 
1 start Firefox. 
2 go to any AMO
3 open new tab and go to about:home
4 switch to first tab
5 let android kill firefox by using another app
6 switch back
7 long tap a link.
effect: zh-cn in menus 
Side effect: the pref is now zh-cn instead of tw until i quit firefox