Last Comment Bug 677919 - Inconsistent rendering of Chinese characters
: Inconsistent rendering of Chinese characters
Status: NEW
:
Product: Core
Classification: Components
Component: Layout: Text (show other bugs)
: 8 Branch
: x86 Windows 7
: -- normal with 9 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
Mentors:
Depends on: 729982
Blocks:
  Show dependency treegraph
 
Reported: 2011-08-10 08:43 PDT by nullpointer
Modified: 2013-12-18 01:05 PST (History)
16 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Screenshot showing the issue. Rendering inconsistencies have been highlighted. (241.08 KB, image/png)
2011-08-10 08:43 PDT, nullpointer
no flags Details
testcase1 (2.87 KB, text/html)
2011-08-11 00:16 PDT, nullpointer
no flags Details
Workaround for web developers (2.88 KB, text/plain)
2011-08-11 00:28 PDT, nullpointer
no flags Details
testcase1 screenshot (232.71 KB, image/png)
2011-08-11 05:42 PDT, nullpointer
no flags Details
testcase2 screenshot (210.62 KB, image/png)
2011-08-11 05:46 PDT, nullpointer
no flags Details
testcase3 - an article from Xinhuanet (3.12 KB, text/plain)
2011-08-12 00:50 PDT, nullpointer
no flags Details
another example of ugly character rendering (693.98 KB, image/png)
2012-02-20 01:46 PST, goldyn chyld
no flags Details
fail case capture (167.71 KB, image/png)
2013-07-05 07:34 PDT, leesei
no flags Details
success case after setting lang="ja" (166.14 KB, image/png)
2013-07-05 07:36 PDT, leesei
no flags Details

Description nullpointer 2011-08-10 08:43:20 PDT
Created attachment 552086 [details]
Screenshot showing the issue. Rendering inconsistencies have been highlighted.

User Agent: Mozilla/5.0 (Windows NT 6.1; rv:8.0a1) Gecko/20110809 Firefox/8.0a1
Build ID: 20110809030751

Steps to reproduce:

I made a list containing Chinese characters with Workflowy (http://workflowy.com)


Actual results:

Firefox rendered the Chinese characters inconsistently, i.e. some characters appeared bolder than others, some other characters were fuzzier than the rest.


Expected results:

Chinese characters should be should rendered in a uniform way.
Comment 1 Makoto Kato [:m_kato] (PTO 6/20-21, 6/24) 2011-08-10 20:47:55 PDT
Kaixi, could you attach HTML file to reproduce this?
Comment 2 nullpointer 2011-08-11 00:10:43 PDT
I've been able to pinpoint the problem I think. When a web page doesn't specify a Chinese font in their CSS rules, Firefox will attempt to render the page with 2 different fallback fonts, Simsun (which is the default Simplified Chinese font in Firefox) and MS PGothic.

In all those cases, Firefox should only use Simsun.
Comment 3 nullpointer 2011-08-11 00:14:29 PDT
I can consistenly reproduce this with a static HTML file that meet the above conditions.
Comment 4 nullpointer 2011-08-11 00:16:58 PDT
Created attachment 552320 [details]
testcase1

HTML file demonstrating the problem. I used the Fontinfo addon to find out the fonts used by Firefox to render the page.
Comment 5 nullpointer 2011-08-11 00:28:47 PDT
Created attachment 552322 [details]
Workaround for web developers

A workaround for web developers is to put a Latin font first and then a Chinese font in their CSS file, i.e. font-family: Arial, Simsun;
Comment 6 kitchin 2011-08-11 04:17:00 PDT
Can you post screenshots from the two test cases?

I don't see any bold characters, but I am on WinXP. Nightly, fresh profile. Fontinfo addon says it is using Arial and SimSun only. Also tested "Namoroka/3.6.21pre" and "Firefox/6.0 SeaMonkey/2.3" and saw no bold.
Comment 7 nullpointer 2011-08-11 05:42:44 PDT
Created attachment 552345 [details]
testcase1 screenshot

Here's the screenshot of testcase1.
Comment 8 nullpointer 2011-08-11 05:46:34 PDT
Created attachment 552346 [details]
testcase2 screenshot

Here's the screenshot of what the page looks like after I change 

body { font-family: Arial; } 

to 

body { font-family: Arial, Simsum; }
Comment 9 nullpointer 2011-08-11 05:50:01 PDT
btw, I am on Windows 7 Professional (English Edition). And neither Chrome nor IE9 have any problems rendering Chinese characters.
Comment 10 nullpointer 2011-08-11 05:56:48 PDT
I just tested everything with a fresh profile on the latest Nightly 32-bit build and the problem is still the same.
Comment 11 kitchin 2011-08-11 06:14:31 PDT
I do not have the font "MS PGothic" on my U.S. WinXP so that is why I do not see the bug.

Microsoft says:

Products that supply this font
Office Mac 2008	5.02
Windows 7	5.01
Windows Server 2008	5.00
Windows Vista	5.00

http://www.microsoft.com/typography/fonts/font.aspx?FMID=1271
Comment 12 nullpointer 2011-08-11 09:38:28 PDT
(In reply to kitchin from comment #11)
> I do not have the font "MS PGothic" on my U.S. WinXP so that is why I do not
> see the bug.

I've uploaded the MS PGothic font file to my Dropbox account. Could you download the font, install it on your system and test again?

http://dl.dropbox.com/u/1829895/msgothic.ttc
Comment 13 kitchin 2011-08-11 15:19:03 PDT
Confirming bug. After I installed the "MS PGothic" font to WinXp, the test page is displayed in a mixture of "MS PGothic" and "SimSun", resulting in some bold-looking characters, as in the screenshot. Is it due to the fact that Japanese uses a subset of Chinese characters? I think "MS PGothic" is a Japanese font.

Bug:
Firefox Nightly
Seamonkey 2.3 Beta (Firefox/6.0 SeaMonkey/2.3)
Namoroka/3.6.21pre

No bug:
IE8, latest release
Chrome, latest release
Opera, latest release
Comment 14 Jonathan Kew (:jfkthame) 2011-08-11 15:45:35 PDT
The test page has no indication of language; hence Firefox can't tell whether to default to the Japanese or Chinese (which locale?) fonts from preferences (for the CJK characters not supported by Arial, the font actually specified by the page). In my testing (on a US English system), it appears to give priority to the Japanese font prefs; but then if there are specific characters not supported in the default Japanese font (MS PGothic), it falls back to the Chinese one (SimSun).

This prioritization should depend on the browser's Accept-Languages settings, and on the user's system locale. If Chinese is given priority in Accept-Languages, for example, I think this should resolve the problem. (You need to restart the browser after adding to the list of "preferred languages" in Tools/Options/Content.)

Also, if you add a language tag such as  lang="zh" to the <body> element, the CJK characters all render with SimSun, as this provides the hint needed to tell the browser which font preference to use.

It's not clear to me how Firefox should be expected to guess the "best" font to use when no appropriate style or language information is available, however.
Comment 15 nullpointer 2011-08-11 15:57:45 PDT
Thanks for the explanation. I deliberately left out any language tags in the test cases. Why can't Firefox guess the best font to use if both Chrome and IE can?
Comment 16 kitchin 2011-08-11 20:01:26 PDT
Bisecting, Firefox 1.0 also has this bug.

Also, if I paste the MS PGothic text from Firefox Nightly to WordPad, it is converted to this:
Font [SimSun] Font Size [10] Font Script [Chinese_GB2312]

If I highlight the text in WordPad and try to change it to "MS PGothic," I cannot, unless I first change Font Script to "Western":
Font [MS PGothic] Font Size [10] Font Script [Western]

The fact that Opera, Chrome and IE agree with each other makes me think the WinAPI is being used differently. Perhaps alpha order plays a role, rightly or wrongly.

The alternative is that Opera, Chrome and IE are all using heuristics, to survey the characters. To test that, we need a test case with only the bold characters. Certainly Chrome uses heuristics for some purposes, because it offers to translate the document from Chinese (Simplified Han) to English!
Comment 17 kitchin 2011-08-11 20:25:21 PDT
See also:
Bug 543200 font fallback should try to use the same font for a complete character cluster or word
Comment 18 nullpointer 2011-08-12 00:12:14 PDT
(In reply to Jonathan Kew from comment #14) 
> This prioritization should depend on the browser's Accept-Languages
> settings, and on the user's system locale. If Chinese is given priority in
> Accept-Languages, for example, I think this should resolve the problem. (You
> need to restart the browser after adding to the list of "preferred
> languages" in Tools/Options/Content.)
> 
> Also, if you add a language tag such as  lang="zh" to the <body> element,
> the CJK characters all render with SimSun, as this provides the hint needed
> to tell the browser which font preference to use.

Adding the language tag definitely makes Firefox render everything with SimSun but adding Chinese in Preferred Languages doesn't resolve the problem.
Comment 19 nullpointer 2011-08-12 00:50:31 PDT
Created attachment 552615 [details]
testcase3 - an article from Xinhuanet

This is yet another testcase, which is an article from today's news on Xinhuanet (http://news.xinhuanet.com/photo/2011-08/12/c_121848818.htm). I copied the contents of the article to an html file which doesn't has any lang tag or css rule for Chinese characters. 

Here Firefox uses 3 fallback fonts to render Chinese characters, Gulim, Simsun and MS PGothic.
Comment 20 Virtual_ManPL [:Virtual] - (ni? me) 2011-08-12 03:25:03 PDT
confirmed on
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:8.0a1) Gecko/20110812 Firefox/8.0a1
Comment 21 Fanolian 2011-08-28 11:47:12 PDT
I see a different scenario.

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0a1) Gecko/20110828 Firefox/9.0a1

In a new profile, extension Fontinfo is showing that testcase1 (sans workaround) and testcase3 are using only Arial (for English characters and numbers) and MingLiu_HKSCS (for Chinese characters, Traditional or simplified) on my machine. No Gulim, Simsun or MS PGothic are involved. I had all these fonts installed.
On the page in xinhuanet, my Nightly is using Mingliu_HKSCS and Arial on the first 3 paragraphs, and Simsun only for the remaining paragraphs.


My Windows 7 Sp1 English version settings FWIW:
Region and Language-
Format: Chinese (Traditional, Hong Kong S.A.R.)
Location: Hong Kong S.A.R.
Language for non-Unicode programs: Chinese (Traditional, Hong Kong S.A.R.)

Also My Nightly's chrome's font follows the setting of "Fonts for Traditional Chinese (Hong Kong)" in Options > Content > Font, which may differ from you guys.
Comment 22 goldyn chyld 2012-02-20 01:46:10 PST
Created attachment 598799 [details]
another example of ugly character rendering
Comment 23 goldyn chyld 2012-02-20 01:46:46 PST
Comment on attachment 598799 [details]
another example of ugly character rendering

I have found this inconsistent rendering of Chinese characters frustrating for quite some time now. Finally I decided to register here, on Bugzilla in hoping you could fix this.

Why can Chrome or Safari display them perfectly, yet Firefox keeps having issues? (Mac user here.)

I believe the default rendering font for Chinese characters should be Heiti SC - I've altered my Firefox' userChrome.css to make the browser use this font when displaying Chinese characters in the tabs (since Chinese characters look uneven, ugly in tabs too, by default), and it works like a charm.

Here's what I use in the userChrome.css (through Stylish):

{font-family: "Lucida Grande", "Heiti SC" !important;}


I wish something like this could be done for text rendering on websites, too. Please, do something about it!
Comment 24 Gen Kanai [:gen] 2012-02-20 18:50:30 PST
(In reply to Jonathan Kew (:jfkthame) from comment #14)

> It's not clear to me how Firefox should be expected to guess the "best" font
> to use when no appropriate style or language information is available,
> however.

Can we do what Safari and or IE does in the same situation?
Comment 25 Masatoshi Kimura [:emk] 2012-02-20 21:36:07 PST
(In reply to Kaixi Luo from comment #15)
> Thanks for the explanation. I deliberately left out any language tags in the
> test cases. Why can't Firefox guess the best font to use if both Chrome and
> IE can?
Because it is not the "best" for Japanese text. We, Japanese users, sometimes suffered from ugly rendering of IE and Chrome. There's no solution everybody can be comfortable because of Han Unification. It's the reason authors should specify the language explicitly.

(In reply to Gen Kanai [:gen] from comment #24)
> Can we do what Safari and or IE does in the same situation?
Things are not so easy.
Comment 26 Masayuki Nakano [:masayuki] (Mozilla Japan) 2012-02-20 22:22:19 PST
(In reply to Masatoshi Kimura [:emk] from comment #25)
> (In reply to Gen Kanai [:gen] from comment #24)
> > Can we do what Safari and or IE does in the same situation?
> Things are not so easy.

Yeah. As jfkthame said in comment 14, we're using Accept-Language pref values for deciding the priority between CJKT languages. Normally, users must download their language's localized build or customize Accept-Language pref if they don't use the localized build. If so, the list gives priority for the language. I guess that you're using English Nightly build on non-CJKT Windows and you have never customized the Accept-Language list. If so, we try Japanese, Korean, Chinese Simplified, Chinese HK, Chinese Taiwan by this order.
http://mxr.mozilla.org/mozilla-central/source/gfx/thebes/gfxPlatform.cpp#1014
Comment 27 Masatoshi Kimura [:emk] 2012-02-21 00:29:42 PST
(In reply to Masayuki Nakano (:masayuki) (Mozilla Japan) from comment #26)
> I guess that you're using English Nightly build on
> non-CJKT Windows and you have never customized the Accept-Language list.
Per comment #18, Accept-Language settings do not have an effect (and I confirmed).
Our fallback order selection may have a bug.
Comment 28 Masatoshi Kimura [:emk] 2012-02-21 06:03:03 PST
I found [Regional and Language Options] > [Formats] virtually took over the settings from Accept-Languages. Is this intentional?
Even if it's deliberate, I'm not sure it's correct to obtain the default language from the [Format] settings for the purpose of font selection. Using [Language for non-Unicode programs] would be more appropriate (and what other browsers did).
Comment 29 Masatoshi Kimura [:emk] 2012-02-21 06:28:30 PST
If the font-family property does not list a generic family, it would be appended. The appended family would be taken from the "font.name.variable.<language>" pref.
http://hg.mozilla.org/mozilla-central/file/8822243a8d6c/layout/style/nsRuleNode.cpp#l2749
nsStyleFont::Init would call nsPresContext::GetLanguageFromCharset() to get the language if the language is not specified on the Web page explicitly.
http://hg.mozilla.org/mozilla-central/file/8822243a8d6c/layout/style/nsStyleStruct.cpp#l158
nsPresContext would get the mLanguage from nsLanguageAtomService::GetLocaleLanguage().
http://hg.mozilla.org/mozilla-central/file/8822243a8d6c/layout/base/nsPresContext.cpp#l1129
nsLanguageAtomService::GetLocaleLanguage() would call nsLocaleService::GetApplicationLocale().
http://hg.mozilla.org/mozilla-central/file/8822243a8d6c/intl/locale/src/nsLanguageAtomService.cpp#l113
On Windows, nsLocaleService would get the application locale from GetUserDefaultLCID() which corresponds to [Formats] settings.
http://hg.mozilla.org/mozilla-central/file/8822243a8d6c/intl/locale/src/nsLocaleService.cpp#l150
Comment 30 Masatoshi Kimura [:emk] 2012-02-21 06:52:05 PST
If the application locale is not CJK, "font.name.*.<lang>" and "font.name-list.*.<lang>" do not contain the CJK fonts, hence expanded generic family also do not.
So Accept-Language settings would have an effect.
If the application locale is Japanese, "MS PGothic" would have precedence over "SimSun" regardless of Accept-Language settings because "font.name.*.ja" contain the former.
If the application locale is Chinese, "SimSun" would have precedence regardless of Accept-Language settings.
Comment 31 goldyn chyld 2012-02-22 00:30:10 PST
Guys, is there a way Firefox could utilize Heiti SC font to display Chinese characters? This font seems to work best, at least for the Chinese characters in tabs...
Comment 32 Masatoshi Kimura [:emk] 2012-02-22 02:41:11 PST
(In reply to goldyn chyld from comment #31)
> Guys, is there a way Firefox could utilize Heiti SC font to display Chinese
> characters? This font seems to work best, at least for the Chinese
> characters in tabs...
Please do one of the following:
- Set the system locale to Chinese (But I don't know how to change the system locale on Mac OS X).
- If the system locale is non-CJK and you do not want change it, add Chinese to "preferred languages" preference of Firefox.
Comment 33 goldyn chyld 2012-02-22 14:30:51 PST
Ok, wow this seems to have done the trick! I moved Chinese/China [zh-cn] language on top of my preferred languages in Firefox and it looks like the Chinese characters appear even now!

Thanks!
Comment 34 goldyn chyld 2012-02-23 11:26:49 PST
It seems to be working fine now... Would still be nice if it worked this good without changing my default language to Chinese (on Firefox), too...
Comment 35 Masatoshi Kimura [:emk] 2012-02-23 18:00:59 PST
There's no way to determine whether the character is indeed Chinese because Unicode uses the same code-points for Chinese, Japanese, and Korean Ideographs. Chrome and IE just assume that everything is Chinese if language-info is unavailable.
While it may be possible to guess the language from the content (like universalchardet), it is not an easy task.
Comment 36 goldyn chyld 2012-02-26 04:58:46 PST
Why can't Firefox also "just assume that everything is Chinese if language-info is unavailable" then?
Comment 37 Masatoshi Kimura [:emk] 2012-02-26 05:04:02 PST
See comment #25.
Comment 38 goldyn chyld 2012-02-26 05:05:26 PST
I see... So for now the only solution to solve the ugly rendering of Chinese characters is to set Firefox "preferred language" to Chinese?
Comment 39 Masatoshi Kimura [:emk] 2012-02-26 05:12:24 PST
(In reply to goldyn chyld from comment #38)
> I see... So for now the only solution to solve the ugly rendering of Chinese
> characters is to set Firefox "preferred language" to Chinese?
Unfortunately, yes. Sorry for the inconvenience to Chinese users.

I have one question. Do Chinese users usually use English version of Firefox? Chinese version sets the "preferred language" to Chinese by default.
Comment 40 goldyn chyld 2012-02-26 05:14:31 PST
It's more of an inconvenience for us "non-Chinese" users, since we generally don't use the Chinese version of Firefox...
Comment 41 Masatoshi Kimura [:emk] 2012-02-26 05:39:07 PST
Do "non-Chinese" users usually see Chinese texts?

BTW I recalled another work around.
1. Open about:config.
2. Select New > String from the context menu.
3. Enter "font.name-list.serif.x-western" as the preference name (assuming you usually set "preferred language" to English).
4. Enter "Heiti SC" as the value.
5. Restart Firefox.

This work around would not work on CJK versions of Operating systems until bug 729982 is fixed.
Comment 42 goldyn chyld 2012-02-26 05:53:45 PST
Wow, this worked!! (I'm on a Mac) -- thanks so much! :)
Comment 43 goldyn chyld 2012-02-27 06:45:43 PST
Masatoshi Kimura, I've sent you an e-mail... :)
Comment 44 leesei 2013-07-05 07:34:13 PDT
Confirmed here using English Firefox 22 in Linux Mint 15 Cinnamon.
I've tried @Kimura's suggestion of adding font.name-list but the issue remains.
Must "Heiti SC" match a font name on my system? Is there a more general setting?

When I add lang="ja" or lang="zh" to the <html>, the font rendering is okay.
See the attached pngs.

Again, setting preferred language in preference won't work (bug 729982).
Comment 45 leesei 2013-07-05 07:34:57 PDT
Created attachment 771630 [details]
fail case capture
Comment 46 leesei 2013-07-05 07:36:11 PDT
Created attachment 771631 [details]
success case after setting lang="ja"
Comment 47 Masatoshi Kimura [:emk] 2013-07-05 08:38:16 PDT
The font name depends on your Operating System.
Please copy the font name from your Chinese settings. (For example, see the value of "font.name-list.serif.zh-CN" in about:config.)
Comment 48 leesei 2013-07-05 09:52:28 PDT
Thanks for the reply.

I cannot find any other "font.name-list" in my settings.
I tried to use "WenQuanYi Micro Hei" (hinted from FontInfo and font conf) but test1 is still ugly :-(.

http://i74.photobucket.com/albums/i261/leesei/screenshot/Screenshotfrom2013-07-06004552.png

Note You need to log in before you can comment on or make changes to this bug.