352757 - Non-Latin glyphs rendered as ? when displaying UTF-8 encoded messages, and also on some web pages

Status: RESOLVED WORKSFORME

(Core Graveyard :: GFX: OS/2, defect)

Description

[Alex Taylor]

User-Agent:       Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:1.8.0.5) Gecko/20060729 SeaMonkey/1.0.3
Build Identifier: Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:1.8.1b2) Gecko/20060913 SeaMonkey/1.1b

When reading newgroup messages encoded in UTF-8 (with any transfer-encoding) in the mail/news message viewer, certain characters from outside the ISO-8859-1 charset do not display correctly.  Instead, they display as question marks. 

Oddly, these glyphs DO appear correctly in the "view message source" window.

I see this particularly with Japanese text in UTF-8 messages.  Japanese 
characters are rendered properly under other encodings such as ISO-2022-JP; the 
problem appears to be specific to UTF-8.


A similar problem appears on some web pages (within the browser) which use UTF-8 encoding.  However, it does not affect all such web pages (whereas it DOES seem to affect all UTF-8 messages).

As one example, many Greek characters on the page http://en.wikipedia.org/wiki/Zeta_%28letter%29 appear as question marks.  Again, the characters appear correctly under the "view source" window.


This problem affects multiple versions of Mozilla, Thunderbird and SeaMonkey 
running under OS/2.  Other versions tested (including but not limited to) are SeaMonkey 1.0.4, Thunderbird 1.5, Mozilla 1.7.12, and IBM Web Browser 2.05.  It is also not specific to any one system: it is 100% reproducible on all OS/2 systems I have tested with.  


If I understand correctly, the correct glyphs are supposed to be substituted into the text from the font(s) configured in "Other Languages".  For some reason, however, this substitution does not seem to be taking place (except in the source viewer window).

Reproducible: Always

Steps to Reproduce:
1. Open the the Mail and News component.

2. Go into a Newsgroup account, and open the newsgroup "sci.lang.japan".

3. Read through messages, either in the integrated message window or the standalone message window, until you locate one which uses a UTF-8 charset encoding, and contains Japanese text.  (Example message-ID: "1cpbvozmdwu8f.1ioo2s1rkqgsk$[e-mail]")

4. Japanese characters in message appear as question marks "?" instead of rendering correctly.

5. From the "View" menu, select "Character Encoding".  Verify that "Unicode (UTF-8)" is selected.  Characters still display as question marks.

6. From the "View" menu, select "Message Source".  Japanese characters appear correctly in the message source window.
Actual Results:  
Japanese characters display as question marks in the message window, and as the correct glyphs in the source window.

Expected Results:  
Japanese characters should display as the correct glyphs in the message window.


Some configuration notes for my main testing system follows.

Font configuration for 'Other Language':
- Proportional: Serif
- Serif:        Times New Roman MT 30
- Sans-serif:   Arial Unicode MS
- Monospaced:   Monotype Sans Duospace WT J

The fonts selected for 'Western' encoding vary, and do not appear to have any effect on the problem whatever they are set to.  (The Western monospaced font is usually "Courier New", although I have also set it to "Monotype Sans Duospace WT J".)

CONFIG.SYS has:
  COUNTRY=001,C:\OS2\SYSTEM\COUNTRY.SYS
  SET LANG=en_CA
  SET MOZILLA_USE_EXTENDED_FTLIB=T

The codepage setting is "CODEPAGE=850,932"; this varies on other systems where the problem can be reproduced, but is generally one of "850,437", "850,1004" or "850,932".

The Innotek Font Engine 2.60B is installed, and the top-level key exists under 
both HKEY_CURRENT_USER and HKEY_LOCAL_MACHINE (most of the actual configuration 
is only in the latter category).

Some Mail and News preferences of possible interest:
- Open messages in a new window
- Default character encoding: ISO-8859-1
- "Apply default to all messages" is NOT enabled
- Use the following font width: Fixed (but I have also tried Variable)

Comment 1

[Peter Weilbacher]

While everything works as expected on my system, several people reported that it happens for them, confirming.

I am not sure if you said that on the newsgroups, but you probably tested with a fresh profile, right? Or with a fresh one where you just adapt the unicode fonts?

Comment 2

[Alex Taylor]

Attachment: Screenshot showing sample message in message window

Here's what a sample message looks like.  The ?'s should be Japanese characters.

Comment 3

[Alex Taylor]

Attachment: Screenshot showing sample message in source viewer

Here's a screen capture of the same sample message as seen in the "view message source" window.  You can see what the glyphs are supposed to look like.

Comment 4

[Alex Taylor]

(In reply to comment #1)
> I am not sure if you said that on the newsgroups, but you probably tested with
> a fresh profile, right? Or with a fresh one where you just adapt the unicode
> fonts?

I've tried with new profiles.  In fact, I just tried it right now (in Thunderbird 1.5, this time).  The problem remains the same.

Comment 5

[Mike Cowperthwaite]

Could you save the message itself as a .EML file and also attach it to this bug?

Comment 6

[Alex Taylor]

Attachment: Sample message exhibiting problem

Here's a message I saved a while ago.  It exhibits the problem on my system(s).  1) Open SLJ.EML file in SeaMonkey.
2) Go to "View" -> "Character Encoding" and select "UTF-8".

Comment 7

[Martin Sch]

I see the same problem.

Comment 8

[Mike Cowperthwaite]

(In reply to comment #0)
> it is 100% reproducible on all OS/2 systems I have tested with.  
> 
> If I understand correctly, the correct glyphs are supposed to be substituted
> into the text from the font(s) configured in "Other Languages".  For some
> reason, however, this substitution does not seem to be taking place (except in
> the source viewer window).

CC'ing the only person I know of with OS/2 experience.  I recently read a comment, in a TB newsgroup I think, stating that Unicode under OS/2 did not work as smoothly as it could.

Bug 189480 was similar but fixed a couple years ago.  See also bug 347503, altho I see a similar symptom, with the .EML attached to this bug, under Win2K 
-- but that's in the titlebar, not the message body.

Possible avenue of exploration: in the Fonts dialog, you can set the desired font for each language group.  Do you have the same font specified for Unicode that you do for Japanese?  If not, does changing that font make this work?  Of course, that might break other Unicode messages using a different alphabet.

Comment 9

[Peter Weilbacher]

Alex, I just finished building a debug build to maybe test this with you at WSE but just now I saw that you said that your setup has
   SET MOZILLA_USE_EXTENDED_FTLIB=T
while you really have to use
   set MOZILLA_USE_EXTENDED_FT2LIB=T
Note the extra "2" in FT2LIB. I hope that this makes the difference!

I think it would really be time to stop requiring this variable, and make it default. The original bug says that this was just for the first trunk nightlies, but somehow it stayed around...

Comment 10

[r.navalinskas]

Attachment: Incorrect rendering of fonts

I am not sure whether it is the same problem, but looks like it - fonts are rendered incorrectly

Comment 11

[Peter Weilbacher]

(In reply to comment #10)
> I am not sure whether it is the same problem, but looks like it - fonts are
> rendered incorrectly

As you seem to be running Windows _and_ you are seeing the problem on a non-UTF-8 site your problem is certainly not related at all.

Comment 12

[Peter Weilbacher]

(In reply to comment #9)
> but just now I saw that you said that your setup has
>    SET MOZILLA_USE_EXTENDED_FTLIB=T
> while you really have to use
>    set MOZILLA_USE_EXTENDED_FT2LIB=T
> Note the extra "2" in FT2LIB. I hope that this makes the difference!

It was probably just a typo in the bug report. I checked Alex' setup on Warpstock and could not find any difference with respect to mine that would explain why this happens.

To further check if all requirements for extended font display are there, I created a small command line app from the Mozilla source code:
- libc05 version (for comparison with SeaMonkey 1.0.x and Firefox 1.5.x): 
     http://weilbacher.org/Mozilla/ft_func_test_libc05.exe
- libc061 version (for comparison with trunk builds):
     http://weilbacher.org/Mozilla/ft_func_test_libc061.exe
- source code: http://weilbacher.org/Mozilla/ft_func_test.c
Anybody who has the problem that Alex described in comment 0, please download the program and run it in the same environment as the Mozilla product that shows the problems, and pass the name of the executable on the command line. E.g. to find a problem with the SeaMonkey setup run
   ft_func_test_libc05.exe seamonkey.exe
It should then print if your setup supports the extended font functions.

Comment 13

[Alex Taylor]

I think I've finally nailed it!

For glyph display to work properly, BOTH of the following must be true:

1. SET MOZILLA_USE_EXTENDED_FT2LIB=T must be set in the environment; AND

2. HKEY_CURRENT_USER\SOFTWARE\InnoTek\InnoTek Font Engine\Applications\SeaMonkey, Enabled=0x00000001 must be set in the registry.

That second one is what I was missing on my laptop (and, I'm a bit embarrassed to say, the first is what I was missing on my desktop).  I only ever installed Font Engine 2.60, so all the FT2LIB configuration was under HKEY_LOCAL_MACHINE instead of HKEY_CURRENT_USER.  

Now, I HAD (based on Andy's advice) created the key HKEY_CURRENT_USER\SOFTWARE\InnoTek\InnoTek Font Engine\AntiAliasing, but never thought to create Applications\SeaMonkey under there as well (because of course it's defined under the HKEY_LOCAL_MACHINE tree).

I'm going to do some wider testing, but I seem to have the correct display of Japanese (and Korean, and Chinese) glyphs on both laptop and desktop working for the time being.

Comment 14

[Peter Weilbacher]

Alex, Andy, I am a bit confused about the FT2LIB settings in the registry. Do we need to modify the GFX code to look at the other one, too?

Comment 15

[Andy Willis (abwillis)]

Have you tried Trunk (nightly)?  The HKCU should not be required on it at all (with the 2.6 version of ft2lib)... if it is then I missed something in my patch.  I am fairly certain that it was not checked into branch. http://lxr.mozilla.org/seamonkey/source/gfx/src/os2/nsGfxFactoryOS2.cpp#95
This is where the code starts.
http://lxr.mozilla.org/seamonkey/source/gfx/src/os2/nsGfxFactoryOS2.cpp#165
Looking at this I suspect I am missing something at this line.

Comment 16

[Peter Weilbacher]

(In reply to comment #15)
> Have you tried Trunk (nightly)?  The HKCU should not be required on it at all
> (with the 2.6 version of ft2lib)... if it is then I missed something in my
> patch.  I am fairly certain that it was not checked into branch.

That was the patch from bug 349439 which is indeed missing from branch where we should add it. With the new "NPOTB" policy I think I can do that tonight.

Comment 17

[Andy Willis (abwillis)]

I just double checked this on Trunk and it isn't right... I did still have Seamonkey enabled under HKCU and it was working... I set it to 0 and I got question marks so I missed something there.  I will have to look into it further.

Comment 18

[Andy Willis (abwillis)]

Ok, bad check... if HKCU has Seamonkey.exe then it has to be enabled set to 1.  I removed Seamonkey.exe entirely from HKCU and then it read the one from HKLM and worked fine so this code will honor HKCU over HKLM (which on a multi-user environment would be correct though the way that ft2lib actually works really isn't). I would say a readme entry and leave it.

Comment 19

[Peter Weilbacher]

What text do you suggest for the README? It currently says

   - set MOZILLA_USE_EXTENDED_FT2LIB=T
     If you have the Innotek Font Engine installed this variable enables
     special functions in SeaMonkey to handle unicode characters.

I think we could add

     Make sure to use either use the HKEY_LOCAL_MACHINE or the
     HKEY_CURRENT_USER tree in the registry but not both and confirm that
     the application is enabled for Font Engine functionality.

But really, such things belong into the Font Engine README and not the Mozilla one. And it's not entirely accurate, either. (To make it accurate, I suppose we would have to come up with an even longer text?!)

Comment 20

[Andy Willis (abwillis)]

The Innotek readme isn't the place for it as we are using ft2lib programatically and their readme doesn't take that into account.  Ft2lib v2.4 only sees HKCU and v2.6 only sees HKLM (which their readme does address the right one to use)... we are having to look for both as we don't know which the user has.  We really only have a problem if the user has 2.6 and has HKCU with a disabled mozilla in it.  If it is enabled or non-existent in HKCU then it won't hurt us.  
I think we could add:

     Make sure to either use the HKEY_LOCAL_MACHINE or the HKEY_CURRENT_USER
     tree in the registry as appropriate for your version of ft2lib as per 
     its readme. If you have both, make sure that it the Mozilla product is
     either not in HKEY_CURRENT_USER  or at least not disabled in
     HKEY_CURRENT_USER if using ft2lib v2.6.

Comment 21

[Peter Weilbacher]

I asked in .l10n and was told that even README.txt is frozen on the branches, so we don't need to discuss additions to it further for now (I will try to remember to add your paragraph for my unofficial builds). I still have to rewrite the font stuff for Thebes but I am pretty certain that MOZILLA_USE_EXTENDED_FT2LIB will be gone with that.

As it works for Alex with the correct setup I am marking this WFM.

Comment 22

[Alex Taylor]

(In reply to comment #14)
> Alex, Andy, I am a bit confused about the FT2LIB settings in the registry. Do
> we need to modify the GFX code to look at the other one, too?
 
I can't speak for trunk, but I can say with confidence that under Seamonkey 1.0x (and Firefox/Thunderbird 1.5x), seamonkey.exe (or firefox.exe/thunderbird.exe) MUST be set under HKCU -- whether or not it is also set under HKLM.  (I have both now, and everything works.)

In other words, in order to fix this, I have gone through all my systems (real and virtual) and made sure my registry contains:

HKCU\SOFTWARE\InnoTek\InnoTek Font Engine\AntiAliasing\Enabled=1
HKCU\SOFTWARE\InnoTek\InnoTek Font Engine\Applications\seamonkey.exe\Enabled=1
HKLM\SOFTWARE\InnoTek\InnoTek Font Engine\AntiAliasing\Enabled=1
HKLM\SOFTWARE\InnoTek\InnoTek Font Engine\Applications\seamonkey.exe\Enabled=1
HKLM\SOFTWARE\InnoTek\InnoTek Font Engine\<other settings required by 2.6>

This (plus of course SET_MOZILLA_USE_EXTENDED_FT2LIB=T) causes everything to work.

Of course, the reason I suddenly hit this problem (about a year ago) is that I switched to a new PC... on which I installed FT2LIB 2.6 directly, and never installed 2.4... thus the HKCU keys were never created. 

Comment 23

[siamaketezadi]

I have this problem on my Mac os 10.4 I This site says it is resolved !

Comment 24

[Andy Willis (abwillis)]

This bug would not affect Mac builds.  If there is not a similar bug for Mac then one would need to be opened.  This bug was specific to OS/2 builds.