1892363 - Simplified Chinese Kanji font appears instead of Japanese Kanji font on some websites depending on Accept-Language

Status: RESOLVED FIXED

(Core :: Layout: Text and Fonts, defect)

Description

[u753259]

User Agent: Mozilla/5.0 (Linux; Android 14; Mobile; rv:127.0) Gecko/127.0 Firefox/127.0

Steps to reproduce:

Nightly and Beta versions of Android Firefox display simplified Chinese kanji fonts instead of Japanese kanji fonts on some sites (github, bugzilla, chatgpt, etc.).
This did not happen in previous versions or in other browsers.
I tried resetting "intl.accept_languages" in about:config and the problem went away until I quit those apps.

Comment 1

[Chris Peterson [:cpeterson]]

Zack, thanks for reporting this bug! Can you please share your Firefox's about:support information in this bug?

What is your Android system locale? What was the value of "intl.accept_languages" before you reset it?

Comment 2

[u753259]

Attachment: about_support.txt

Comment 3

[u753259]

(In reply to Chris Peterson [:cpeterson] from comment #1)

Zack, thanks for reporting this bug! Can you please share your Firefox's about:support information in this bug?

What is your Android system locale? What was the value of "intl.accept_languages" before you reset it?

My Android system locale is Japan. The value of "intl.accept_languages" before resetting is "ja-JP,en-US,zh-Hans-CN,zh-Hant-TW,zh-Hant-HK,zh-CN,zh-TW,zh-HK".

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

The bug has a release status flag that shows some version of Firefox is affected, thus it will be considered confirmed.

Comment 5

[Donal Meehan [:dmeehan]]

:boek next week is the last week of beta for Fx126. We have little time to investigate this.
Could this be triaged and prioritized for investigation?

Comment 6

[Mike Hommey [:glandium]]

My Android system locale is Japanese too. intl.accept_languages was ja-JP here, I tried adding zh-CN, but, none of the sites you mentioned (chatgpt, bugzilla, github) are shown to me in Japanese or Chinese. They're all in English...

Comment 7

[Zac McKenney [:zmckenney]]

:zack I see you have this addon for translating pages: TWP - Translate Web Pages. Would you mind disabling or uninstalling this addon and checking again? The settings for this addon has language selectors for translating webpages in it's settings and I'm wondering if this is actually causing your problems. Additionally, there is a mobile version of this addon called TWP - Translate for Mobile that may work better but I haven't tested.

Comment 8

[u753259]

Sorry, my standard system locale is Japanese, but other languages were also set. All my Android languages are Japanese, English, Simplified Chinese (Mainland China), Traditional Chinese (Taiwan) and Traditional Chinese (Hong Kong).
Kanji has slightly different character forms for the same character code kanji depending on the country and region (you may not know this difference unless you use Kanji on a regular basis).
This bug is that Kanji characters in other regions are displayed even though the language is set to Japanese.
This is a trivial problem, but as I always use Kanji, I feel quite uncomfortable.

Comment 9

[u753259]

I have uninstalled TWP and initialized Firefox, but this bug still occurs.

Comment 10

[Mike Hommey [:glandium]]

Can you give a link to a specific page that shows Japanese written with Chinese characters? A screenshot could be useful too.

Comment 11

[Mike Hommey [:glandium]]

FWIW, after adding simplified Chinese via the Android settings, I now have intl.accept_languages set to ja-JP,zh-Hans-CN,zh-CN, and e.g. Google or Wikipedia are still displayed in proper Japanese.

Comment 12

[Mike Hommey [:glandium]]

So does chatGPT if I talk to it in Japanese.

Comment 13

[u753259]

Attachment: simplified_chinese.png

この画像はintl.accept_languagesをリセットする前のchatgptです。

Comment 14

[u753259]

Attachment: japanese_kanji.png

And this is after the reset.

Comment 15

[u753259]

Ahhh, Sorry, I wrote the sentence before I translated it
"この画像はintl.accept_languagesをリセットする前のchatgptです。" -> "This image shows chatgpt before resetting intl.accept_languages."

Comment 16

[u753259]

I don't get this bug with google or wikipedia on my device.

Comment 17

[Mike Hommey [:glandium]]

Okay, I'm definitely able to reproduce, and it's even reproducible with Android in English

My STR:

• Go to about:config and ensure intl.accept_languages is reset.
• Open, Android settings app, select "System", then "Languages & input", then "Languages"
• Set things up so that you have the following languages:
  • English (United States)
  • 日本語 (日本)
  • 简体中文(中国)
• In Firefox, go to https://chat.openai.com
• As a prompt, type 変化、設置、街角、絵画、抱く、花火

I think the most visible difference to people not accustomed to Chinese/Japanese is in 抱く

If you go back to the Android settings and remove 简体中文(中国), returning to Firefox resets the display (without reloading the page or anything) and switches to the right font.

Comment 18

[Mike Hommey [:glandium]]

I can also confirm this doesn't affect Chrome.

Comment 19

[Mike Hommey [:glandium]]

I can also reproduce with Firefox 125.2.0 (Build #2016016039)

Comment 20

[Chris Peterson [:cpeterson]]

In that case, this bug isn't a regression in Firefox 126, though perhaps it's a regression in 125 or earlier.

Comment 21

[Jonathan Kew [:jfkthame]]

I wonder if this is related to the (fairly recent) implementation of Android font visibility restrictions (anti-fingerprinting) in bug 1826412. Does turning off tracking protection affect the behavior?

Comment 22

[Mike Hommey [:glandium]]

(In reply to Jonathan Kew [:jfkthame] from comment #21)

Does turning off tracking protection affect the behavior?

Assuming I turned off the right thing, no.

Comment 23

[u753259]

The cause may be that Firefox has set only ja-JP as the locale code for Japanese in accept-language.

Firefox for Windows sets ja for Japanese locale code.

Other mobile browsers set both ja and ja-JP.

Adding ja to the beginning of the intl.accept_languages value produces the same result as resetting it.

Comment 24

[Jonathan Kew [:jfkthame]]

(In reply to zack from comment #23)

The cause may be that Firefox has set only ja-JP as the locale code for Japanese in accept-language.

Oh, interesting -- yes, I suspect that could be the issue; the font preferences may not recognize that. If that's the issue, it should be straightforward to fix, I think. I'm out of the office today but will try to check on this when I'm back, if no-one else does it in the meantime.

Comment 25

[Mike Hommey [:glandium]]

14:50.30 INFO: No more integration revisions, bisection finished.
14:50.30 INFO: Last good revision: c55a609359a20d23e6534c7c188007d07bb1877b
14:50.30 INFO: First bad revision: 0f63ab5624960b1aa464bb2e3bd76c6cd196f186
14:50.30 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=c55a609359a20d23e6534c7c188007d07bb1877b&tochange=0f63ab5624960b1aa464bb2e3bd76c6cd196f186

Comment 26

[Chris Peterson [:cpeterson]]

Makoto, this bug is a regression from your Accept-Language change in bug 1847701 in Fx 125. Is this a website/webcompat bug? Or should we change how we format the locale tags in Accept-Language again?

Comment 27

[Jonathan Kew [:jfkthame]]

I suspect (but have not yet confirmed) that this is happening because the GetFontPrefLangFor function called here only recognizes ja, not ja-JP, and we should fix that to make it smarter.

Comment 28

[Mike Hommey [:glandium]]

And ja is not the only locale with the problem:
https://searchfox.org/mozilla-central/source/gfx/thebes/gfxFontPrefLangList.h#1

Comment 29

[Mike Hommey [:glandium]]

It is worth noting that the claim in bug 1847701's commit message that Accept-language is "language-region format like Desktop" is actually not true for Japanese:
https://hg.mozilla.org/l10n-central/ja/file/eaeda87d1d4ecbe97a51beccde04b09e055a5157/toolkit/chrome/global/intl.properties#l23

Comment 30

[Mike Hommey [:glandium]]

Many locales put both "language-region" and "language", but many skip "language-region".

Comment 31

[Makoto Kato [:m_kato]]

Hmm, before landing bug 1847701, accept-language was ja-JP if system locale is Japanese. Chinese was zh-Hans-TW (Taiwan) or zh-Hans-CN (China). But after this change, Chinese becomes zh-TW or zh-CN. It means that font selection in gfx matches with Chinese.

Comment 32

[Makoto Kato [:m_kato]]

(Although I filed bug 1889845, we should normalize this value by Gecko side at future)

Comment 33

[Mike Hommey [:glandium]]

(In reply to Makoto Kato [:m_kato] from comment #31)

Hmm, before landing bug 1847701, accept-language was ja-JP if system locale is Japanese. Chinese was zh-Hans-TW (Taiwan) or zh-Hans-CN (China). But after this change, Chinese becomes zh-TW or zh-CN. It means that font selection in gfx matches with Chinese.

In the geckoview example app, it was indeed ja-JP,zh-Hans-CN with my current Android system settings in version 124. And indeed, if I change that to ja-JP,zh-CN manually, I get the same font selection problem. Seems like comment 27 might be spot on, and we never even matched Japanese before.

Comment 34

[Mike Hommey [:glandium]]

That said, we are also currently not using language-region on desktop.

Comment 35

[Mike Hommey [:glandium]]

This bruteforce approach fixed the issue for me: https://hg.mozilla.org/try/rev/f960adc00da5cea1c5dc9bca5a3c7a3e0903033b, so comment 27 is confirmed. Interestingly, logging the GetFontPrefLangFor function, I can see that it regularly returns Others on my desktop, where it's queried for e.g. ja-jp (lowercase, I haven't looked where it comes from, but it's not from accept_languages).

Comment 36

[Makoto Kato [:m_kato]]

Attachment: Bug 1892363 - Accept-Language shouldn't have country code if language is ja. r=#geckoview-reviewers

Original issue is that font selection doesn't match with Japanese when system
locale is Japanese. Before landing bug 1847701, all far east languages don't
always match. But after langing it, Chinese (Simplified and Traditional) is
macthed only.

Gfx doesn't want that Accept-Langues preference has a country code if
languguage is Japanese or Korean.

So we should return language code only for accept-language if language tag is
ja-JP or ko-* (North or South).

Comment 37

[Jonathan Kew [:jfkthame]]

In principle, this also affects the other font prefs where we use a bare language subtag without region, if the accept-language list (or other source of lang tags) has the language-region form. These include Arabic, Greek, Hebrew, and Thai, as well as Japanese and Korean. In practice those cases are less likely to be noticed, as fallback behavior will likely still pick an appropriate font; there aren't the same region-dependent variants as with CJK. But we should really fix them all.

Comment 38

[Jonathan Kew [:jfkthame]]

Attachment: Bug 1892363 - When looking for font prefs based on a lang tag, try comparing the base lang alone if the whole tag doesn't match. r=#layout-reviewers

Comment 39

[Jonathan Kew [:jfkthame]]

The patch in D208588 is a somewhat ugly hack, but should resolve this AFAICS. We could go the whole way and use Locale parsing and then extract the lang subtag, but that seems overkill given the very limited set of font pref lang codes we're dealing with; and it's my hope that we'll eventually be replacing all this with a more principled lang- and script-driven architecture anyway.

Comment 40

[Makoto Kato [:m_kato]]

I withdraw my fix

Comment 41

[Jonathan Kew [:jfkthame]]

Moving this to Core :: Graphics: Text, as it's really an issue in how gfx/thebes handles lang codes in relation to font prefs, and in principle it could affect other platforms as well, if people modify their accept-language to have <lang-region> codes for cases where gfx only expects the lang.

Comment 42

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1847701

Comment 43

[Jonathan Kew [:jfkthame]]

Actually, this is really about font selection, rather than rendering; -> Layout.

Comment 44

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/943302be4233
When looking for font prefs based on a lang tag, try comparing the base lang alone if the whole tag doesn't match. r=lsalzman

Comment 45

[Jonathan Kew [:jfkthame]]

Attachment: Bug 1892363 - When looking for font prefs based on a lang tag, try comparing the base lang alone if the whole tag doesn't match.

Original Revision: https://phabricator.services.mozilla.com/D208588

Comment 46

[Jonathan Kew [:jfkthame]]

Attachment: Bug 1892363 - When looking for font prefs based on a lang tag, try comparing the base lang alone if the whole tag doesn't match.

Original Revision: https://phabricator.services.mozilla.com/D208588

Comment 47

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: Japanese users may see incorrect (Chinese) font; potentially other similar impacts
• Code covered by automated testing: yes
• Fix verified in Nightly: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: n/a
• Risk associated with taking this patch: low
• Explanation of risk level: simple check for additional forms of language code
• String changes made/needed: none
• Is Android affected?: yes

Comment 48

[Phabricator Automation]

release Uplift Approval Request

• User impact if declined: Japanese users may see incorrect (Chinese) font; potentially other similar impacts
• Code covered by automated testing: yes
• Fix verified in Nightly: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: n/a
• Risk associated with taking this patch: low
• Explanation of risk level: simple check for more lang code forms
• String changes made/needed: none
• Is Android affected?: yes

Comment 49

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/943302be4233

Comment 50

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/9127c3adbf51

Comment 51

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-release/rev/53302f85d080

Comment 52

[Takanori MATSUURA]

Just for information:
You can see the differences between the glyphs more easily with the following letters than with the test HTML.
刃 直 骨 海 角 遙 入

Comment 53

[Ryan VanderMeulen [:RyanVM]]

Added to the 125.0.3 relnotes.

Fixed an issue which could cause Japanese users to see incorrect fonts in some situations