1581822 - Uneven weights and fonts displayed for Chinese pages on Firefox desktop

Status: RESOLVED FIXED

(Core :: Layout: Text and Fonts, defect, P3)

Description

[Igeeit]

Attachment: Annotation 2019-09-17 211429.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0

Steps to reproduce:

Go to a Chinese page, e.g. https://duckduckgo.com/?q=%E9%9D%92

Actual results:

Uneven / different fonts and weights on the same line of text. See attachment.

Expected results:

Consistent font, as with other browsers.

Comment 1

[Jonathan Kew [:jfkthame]]

The issue here is that the content on the DDG page isn't tagged with lang=zh (or similar), and so Firefox doesn't know to favor Chinese fonts; it uses the DDG webfont (as font-family: DDG_ProximaNova) for English content, and then fallback kicks in for the CJK characters.

In the absence of a lang attribute to suggest whether to try the default Japanese or Chinese fonts first, Firefox opts for Japanese; but unfortunately the default Japanese font (Yu Gothic) doesn't support some of the Chinese characters here. So then those characters fall back to a different font (Microsoft YaHei, at least on my system), which results in this "ransom-note" appearance.

Looking at gfxPlatformFontList::AppendCJKPrefLangs, it appears Firefox tries several "clues" to determine which CJK font prefs to use: it'll look first at the lang of the page, but if that's not a CJK language, it'll then check the accept-language setting and the browser's application locale to try and guess the user's preferred language. If all else fails, it ends up with a hard-coded default order that puts Japanese first, then Chinese, then Korean.

In the case of the DDG search-results example, the page is tagged with lang=en-US, so that's no help. And I don't have Chinese (or any other CJK lang) in my accept-languages pref, so that's no help either, and my browser is an en-US build. So there are no useful clues, and I end up with the default Japanese-first behavior.

If I add zh-CN to intl.accept_languages, then I get the default Chinese font used consistently for the DDG results page, avoiding the problem here. Of course, it's likely this would give inferior results for some cases where the (untagged) CJK content on the page is in fact Japanese.

For current Windows systems, at least, maybe it would be better to put Chinese ahead of Japanese in the "last resort" order, on the grounds that the default Chinese font seems to have pretty good coverage of characters used in Japanese (including Hiragana, etc), whereas the default Japanese font doesn't handle Chinese well (as seen here).

:m_kato, WDYT? Should we consider reordering the fallbacks here, or is there anything else we could do to improve this behavior?

Comment 2

[Makoto Kato [:m_kato]]

Actually, we respect UI locale and system locale before using // last resort... (the order is same as old gfx.) comment. Current Windows version uses 1 locale only from system language. (Android etc can support multiple system language for CJ glyph). So user uses en-us locale and en-us version of Firefox, you know, it may use JP font at first. Unfortunately, GetUserPreferredUILanguages has a bug that it doesn't return multiple locales correctly (ie https://stackoverflow.com/questions/52849233/getuserpreferreduilanguages-never-returns-more-than-two-languages. I already report it via feedback hub). If GetUserPreferredUILanguages works well at feature version of Windows, we will be able to remove old workaround.

Also, duckduckgo.com has preference for your locale. So if changing language from Browser Preferred Language to specific such as 中文 (中国), accept language is zh-CN. So this issue is resolved.

Comment 3

[Jonathan Kew [:jfkthame]]

Yes, there are a number of levels at which this can be resolved, from lang attributes on the content to the accept-languages setting, browser UI locale, and system locale. We only depend on the "last resort" order of the font prefs in the case where no usable locale hint was found at any of these levels.

But specifically for this last-resort case, I think we should modify the order to put the Chinese font prefs first. The reason for this being that (on both Windows and macOS, at least) the default preference fonts for Japanese have less complete character coverage than the Chinese ones, there's a greater chance of ending up with a mixture of fonts if we prioritize the Japanese prefs.

In other words, the potential downside of displaying Chinese content with the Japanese font prefs prioritized is worse than the downside of displaying Japanese content with the Chinese font prefs prioritized.

(The change should have no significant effect for users running a Chinese-, Japanese, or Korean-locale browser, or on a C/J/K-locale system, where we'll use the locale to decide which font prefs take precedence.)

Comment 4

[Jonathan Kew [:jfkthame]]

Attachment: Bug 1581822 - Try Chinese font prefs before Japanese when no locale hint is found, to reduce chance of ransom-note font mixtures. r=m_kato

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9cf41163400a
Try Chinese font prefs before Japanese when no locale hint is found, to reduce chance of ransom-note font mixtures. r=m_kato,masayuki,emk

Comment 6

[Cristina Coroiu [:ccoroiu]]

Backed out changeset 9cf41163400a (Bug 1581822) on request by jfkthame

Backout: https://hg.mozilla.org/integration/autoland/rev/19b7d80f9e39e41ed2e5d849e64725f03e660fdd

Comment 7

[Jonathan Kew [:jfkthame]]

Argh - right after I queued this in Lando, I remembered it's going to affect a few testcases where CJK characters are used and there's no specific lang tagging or font selection. So backed out for now; I'll check what metadata updates are needed, and then re-land.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a67173efac26
Try Chinese font prefs before Japanese when no locale hint is found, to reduce chance of ransom-note font mixtures. r=m_kato,masayuki,emk

Comment 9

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/a67173efac26

Comment 10

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #23645 (as of Thu, 31 Oct 2019 17:35:06 GMT) ==

Improvements:

80% tp5n main_normal_fileio windows10-64-shippable opt e10s stylo 3,938,067.75 -> 790,374.25

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=23645