202009 - Some Japanese characters in UTF-8 document display as ?

Status: RESOLVED FIXED

(Core :: Layout: Text and Fonts, enhancement, P2)

Description

[Mary Hoetzel]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3) Gecko/20030312
Build Identifier: Mozilla/5.0 (X11; U; AIX 000770094C00; en-US; rv:1.4a) Gecko/20030402

Some Japanese characters in a UTF-8 encoded document display as ? in Mozilla on
AIX. Examples are the characters with the UCS2 representation 0x3231, 0x3322 and
0x334D (the last 2 display in Mozilla on Linux as ? as well).

It seems to me that the document is displayed using the jisx fonts (jisx0201,
jisx0208) and these characters are not part of these fonts. 
These characters are part of the ibm-udcjp fonts.   

One solution therefore would be to enable the ibm-udcjp fonts (see bugzilla 172625).

Another solution would be to enable the ucs2.cjk-japan-0 fonts (and other
ucs2.cjk fonts as well). In AIX, these fonts could be used to display UTF-8
document because the ucs2 fonts are the fonts using for the UTF-8 locales.

I would like to work on the ucs2 font enablement but I'm not sure what this all
involves. I added a nsFontCharSetInfo entity named UTF8JA in sFontMetricsGTK.cpp
and changed the ucs2.cjk_japan-0 setting in gCharSetMap from unknown to
{ "ucs2.cjk_japan-0",  &FLG_JA,  &UTF8JA  }

This makes the ucs2.cjk_japan-0 fonts to be listed in Font Preferences under
Unicode, and they are now part of the list of fonts looked at in GetFontNames in
nsFontMetricsGTK.cpp, but they are discarded because there is no valid
UnicodeEncoder. But in this case, a UnicodeEncoder would not be necessary, right?

Does anybody have a recommendation for me please?


Reproducible: Always

Steps to Reproduce:
1.
2.
3.

Comment 1

[Mary Hoetzel]

Attachment: HTML testcase

Added an UTF-8 encoded HTML page that displays the 3 characters mentioned
above.

Comment 2

[Mary Hoetzel]

Attachment: GIF file with characters as they should appear

Added GIF image of the characters how they should appear.

Comment 3

[Mary Hoetzel]

Attachment: GIF how Mozilla displays the document

Added GIF image of Mozilla displaying the UTF-8 HTML page in the testcase.

Comment 4

[Roland Mainz]

Mary:
Does adding the line...
-- snip --
{ "ucs2.cjk_japan-0",  &FLG_JA,   &ISO106461     },
-- snip --
... to |gCharSetMap[]| help in your case ?

BTW: It would be nice of someone would attach a "ucs2.cjk_japan-0" PCF font to
this bug that we can do some tests with it... :)

Comment 5

[Mary Hoetzel]

Adding &ISO106461 actually helps with these 3 characters.  Thank you!

But, these 3 characters were part of a string I was testing with and another
character in this string now has a problem. This character is the "No" sign and
it is now different after this change.

I will add new attachments with the entire string as testcase and images how
they look before and after the change.  

Comment 6

[Mary Hoetzel]

Attachment: Updated HTML testcase

Comment 7

[Mary Hoetzel]

Attachment: Displayed with Mozilla on AIX when ucs2.cjk-japan-0 = unknown

Comment 8

[Mary Hoetzel]

Attachment: Displayed with Mozilla on AIX when ucs2.cjf-japan-0 = &ISO106461

Comment 9

[Mary Hoetzel]

Attachment: Proposed patch to enable ucs2 fonts

I cannot reproduce the problem with the "No" sign anymore. Maybe something was
corrupted in my profile. 

It works fine now on AIX after applying the code change in the proposed patch.

It doesn't improve Linux (meaning, the last two characters in the updated
testcase are still displayed as ??) which I wouldn't have expected because
there are no ucs2 fonts installed on my Linux system, but it doesn't hurt
anything either.

Comment 10

[Roland Mainz]

Comment on attachment 121275 [details] [diff] [review]
Proposed patch to enable ucs2 fonts

The patch does not patch nsFontMetricsXlib.cpp

I'll file a new patch in a few mins...

Comment 11

[Roland Mainz]

Attachment: Patch for 2003-04-21-08-trunk

Comment 12

[Roland Mainz]

Comment on attachment 121290 [details] [diff] [review]
Patch for 2003-04-21-08-trunk

Requesting r=/sr= ...

For the log:
The patch has one (minor) flaw: Using |ISO106461| for the "*-ucs2.*"-XLFD fonts
is not a perfect solution since the fonts need to be loaded first before
mozilla knows which glyphs are available in the fonts.
However the fix for that would require to provide a map with the info which
glyphs can be found for a specific XLFD and I think we should get this patch
"in" for 1.4b and then work on a patch which provides the matching CCMap for
those XLFD encodings. I'd like to defer that work since it will be a _FAR_
larger patch than this one (which should work without problems - and the perf.
enhancement should be moved to a new bug).

Comment 13

[Simon Montagu :smontagu]

Comment on attachment 121290 [details] [diff] [review]
Patch for 2003-04-21-08-trunk

r=smontagu for the changes to nsFontMetricsGTK.cpp and nsFontMetricsXlib.cpp,
but I am dubious about making dt-interface system-ucs2.cjk_japan-0 the default
unicode font for all users. Does the testcase work without the change to
aix.js?

Comment 14

[Roland Mainz]

Mary:
The following part can't be right:
-- snip --
// Override default Unicode fonts
pref("font.name.serif.x-unicode", "dt-interface system-ucs2.cjk_japan-0");
pref("font.name.sans-serif.x-unicode", "dt-interface system-ucs2.cjk_japan-0");
pref("font.name.monospace.x-unicode", "dt-interface user-ucs2.cjk_japan-0");
-- snip --
This limits the available unicode fonts to japanese glyphs only (assuming
*-ucs2.cjk_japan-0 fonts only contain japanese glyphs). However the issue is
"hidden" by the fact that Mozilla searches other fonts when the default font
does not provide them (the issue will cause that Mozilla will simply ignore the
prefs set in aix.js for unicode pages with non-japanese content).

Comment 15

[Roland Mainz]

BTW: The whole font pref stuff in aix.js and the similar planned idea for a
solaris.js to store font defaults for Solaris is totally wrong - X11 fonts are
set by the server-side - defining platform-specific prefs on the client side
fails in scenarios like when Mozilla is running on a Solaris machine but
displays itself on an AIX Xserver... ;-/

Comment 16

[Mary Hoetzel]

I should have explained the aix.js default setting. Sorry.

ucs2.cjk_japan-0 fonts have all glyphs not just the Japanese and there are more
ucs2.cjk_japan-0 fonts available on AIX than ucs2.cjk-0 which would have been
the other choice for a default. 

If we would have language specific aix.js files, I would make ucs2.cjk_korea-0
the Korean choice etc.  

I tested Western, East European, Cyrillic, Korean, Chinese, Taiwanese, Thai
documents with this default setting and it works fine.

The fix works without the aix.js change, but as it is, ucs2.cjk_japan-0 seems to
me the best default for AIX at this point (it might change in the future).

To Roland's comment #15: The aix.js (and the other js files) are only suggested
defaults that can be overwritten. It seems reasonable to me to have AIX defaults
that assume it runs with an AIX Xserver, and Solaris defaults that assume it
runs on an Solaris Xserver, etc.
 

Comment 17

[Roland Mainz]

Mary:
Can we split the aix.js issue off and move to a seperate bug and checkin the
gfx/-part of the patch for now ?
I may have a solution which works properly in all cases (e.g. even in the case
of a Solaris client vs. AIX Xserver or AIX client+Xserver but Solaris xfs(=X
font server) etc.) ...

Comment 18

[Roland Mainz]

Taking the bug myself (unless someone else disagrees... :) ...

Comment 19

[Simon Montagu :smontagu]

Comment on attachment 121290 [details] [diff] [review]
Patch for 2003-04-21-08-trunk

OK, I am OK with this in the light of comment 16. Roland, feel free to work on
refining the prefs for the case where client and server are different systems
in a new bug.

Comment 20

[Roland Mainz]

Filed bug 202921 ("Default font prefs in aix.js do not work for non-AIX clients
on AIX Xserver") for the aix.js prefs issue...

Comment 21

[Simon Montagu :smontagu]

Fix checked in.