problem in displaying plane 1 characters

VERIFIED FIXED in mozilla0.9.8



17 years ago
15 years ago


(Reporter: shanjian, Assigned: shanjian)


Windows 2000

Firefox Tracking Flags

(Not tracked)



(6 attachments, 3 obsolete attachments)



17 years ago
Frank tang wrote:
  Subject: Re: Strange plane 1 behaviour for Netscape 6.x
  1. we know we have some bug in NCR and UTF-8 conversion. We are fixing
  them now.
  2. we know our linux/window/mac text rending code do not take care of
  Plane 1 yet. We plan to fix the window version in the near future.
  Please contact if you can help (providing
  information where to find a font, how to install, how to test, etc)
  Otto Stolz wrote:

    Tex Texin had written:

      IE likes plane 1 characters only as NCRs (&#xXXXXX;).

    Lars Marius Garshol wrote:

      for Opera 6.0 it makes no difference how you encode the characters.

      They can be NCRs, encoded in UTF-8, or encoded in UTF-16.

    I have tested three browsers with the example page
    which comprises NCRs of plane 1 characters.
    I have used Windows XP Professional Version 5.1 [German] (Build
    2600.xpclient.010817-1148). I had installed the Code2001 font,
    and set the three registry keys described in that example page.
    - MS Internet Explorer 6.0.2600.0000.xpclient.010817-1148 [German]:
      All plane-1 characters are displayed, in the correct writing
    - Opera version 6.0, Build 101 [German] without Java support:
      The plane-1 characters are not correctly displayed;
      rather, a row of boxes is displayed (only recognizable in
      120% zoom, or larger). There are roughly (but not exactly)
      2 boxes per plane-1 character, viz:
    -> 12 boxes, "Aulesi" -> 15 boxes, "Metelis" -> 13 boxes
      (judging from the word separator, which is displayed correctly);
      "Utah","Brigham", "Young", "’]izai", "’]iudangardjai", "’]ize",
      "Gutane", and "Wulfila" all have exactly twice as many boxes as
      there are characters in these words.
      The writing direction is correct.
    - Netscape 6.2 [English]:
      The Deseret characters are replaced with Cyrillic ones;
      the Etruscan and Gothic words are replaced with some
      unrecognicable blots (possibly some glyphs overlayed).
      The writing direction is correct.
    This means that Opera 6.0 does not display plane-1 characters, in
    all environments.
    On the Opera download pages, I could not find anything dubbed "6.01b".
    Best wishes,
      Otto Stolz

James Kass wrote:


  Freeware Plane One font, Code2001.ttf:
  Plane One HTML test pages in NCR format:
  Plane One HTML test pages as UTF-8
  Hope this is helpful.
  With best regards,
  James Kass.

Comment 1

17 years ago
It looks the utf8 to unicode converter is fine for surrogate
unicode to utf8 issue is log as bug 102595


17 years ago
Attachment #63524 - Attachment is patch: false
Attachment #63524 - Attachment mime type: text/plain → image/gif

Comment 6

17 years ago
Created attachment 63526 [details]
correct display of part of view source in


17 years ago
Attachment #63521 - Attachment description: correct display 1 → correct display of


17 years ago
Attachment #63522 - Attachment description: correct display 2 → correct display


17 years ago
Attachment #63524 - Attachment description: correct display 3 → correct display of <file:///D:/code2001/p1exh3.gif>


17 years ago
Attachment #63524 - Attachment description: correct display of <file:///D:/code2001/p1exh3.gif> → correct display of


17 years ago
Attachment #63525 - Attachment description: correct display 5 → correct display of


17 years ago
Attachment #63526 - Attachment description: correct display 6 → correct display of part of view source in

Comment 7

17 years ago
Target Milestone: --- → mozilla0.9.8

Comment 8

17 years ago
Created attachment 64535 [details] [diff] [review]
working patch, (doesn't work yet.)

Comment 9

17 years ago
Created attachment 64884 [details] [diff] [review]
patch ready for r/sr
Attachment #64535 - Attachment is obsolete: true

Comment 10

17 years ago
Created attachment 64908 [details] [diff] [review]
repost patch to correct carriage/return problem in previous patch

This patch has no semantic difference with previous one. The Previous one seems
to have 
some carriage/return misfunction. Repost just for sure.
Attachment #64884 - Attachment is obsolete: true

Comment 11

17 years ago
After this bug is fixed, there is another problem in test page
right to left text was not rendered correctly. Bug 119983 was 
filed for this problem.

Comment 12

17 years ago
+#define CCMAP_PLANE_FROM_SURROGATE(h)  (((PRUint16)(h) - (PRUint16)0xd800) >> 6)

+#define CCMAP_PLANE(u)  ((((PRUint32)(u))>>16)-1)

hum... this is confusing. we have BMP which is Plane 0, and Plane 1-16. but this
macro will return 0 for Plane 1 and 0x0f for Plane 16. Should we  return 1 for
Plane 1 and 0x10 for Plan 16?
Same thing as CCMAP_PLANE_FROM_SURROGATE, they really return CCMAP_PLANE_MINUS_1
instead of CCMAP_PLANE

+  //testing for plane 1
+  for (k = 0x10000; k < 0x20000; k++) {
+    oldb = IS_REPRESENTABLE(aOtherPlaneMaps[k/0x10000-1], k&0xffff);
+    newb = CCMAP_HAS_CHAR_EXT(ccmap, k);
+    NS_ASSERTION(oldb==newb, "failed to generate extension map correctly");
+  }

Why test only plane 1 (0x10000 - 0x1FFFF), how about plane 2-16 ?

r=ftang for changes in gfx/src/windows/nsFontMetricsWin.h, 
intl/lwbrk/src/nsJISx4501LineBreaker.cpp, and htmlparser/src/nsHTMLTokens.cpp

harishd: please r= the change in htmlparser/src/nsHTMLTokens.cpp
we try to land it for m0.9.8 

shanjian : The following characters are in Plane 1
Old Italic
Byzantine Musical Symbols
Musical Symbols
Mathematical Alphanumeric Symbols

the following is in Plane 2:
CJK Unified Ideographs Extension B 
CJK Compatibility Ideographs Supplement

I really care more about Plane 2. Although the summary of this bug said "plane 1
character", the real issue is how we deal with characters in plane 1-16. Please
make sure your code do not ONLY handle palne 1 (as the for loop I found)

Comment 13

17 years ago
Created attachment 64940 [details] [diff] [review]
update patch as suggested by ftang
Attachment #64908 - Attachment is obsolete: true

Comment 14

17 years ago
cc to marc

Comment 15

17 years ago
Harish, could you review the changes in htmlparser/src/nsHTMLTokens.cpp? thanks.

Comment 16

17 years ago
Comment on attachment 64940 [details] [diff] [review]
update patch as suggested by ftang

Attachment #64940 - Flags: review+

Comment 17

17 years ago
Marc, could you sr my patch? thanks.

Comment 18

17 years ago
Comment on attachment 64940 [details] [diff] [review]
update patch as suggested by ftang

Attachment #64940 - Flags: superreview+
How are we handling the fact that we store lots of data in string classes that
contain UCS2 data in the most minimal form, which doesn't allow any plane 1

Comment 20

17 years ago
We are still using UTF16 in every piece of our string. We only convert surrogate pair
to UCS4 when doing font searching. 

Comment 21

17 years ago
fix checked in. 
Last Resolved: 17 years ago
Resolution: --- → FIXED
We've been using UCS-2, as far as I know, not UTF-16.  If we're using UTF-16,
then Length() is not technically the number of PRUnichar units in a string,
although I guess we could get away with only certain bits of code knowing that
there are really fewer characters.

Comment 23

17 years ago
*** Bug 102557 has been marked as a duplicate of this bug. ***
Our code only supports the simplest form of UCS-2, where each 16-bit unit is
seen as one character. Support for UTF-16 (extended UCS-2) would come at the
cost of a runtime hit, or a footprint hit by having 32-bit units, either of
which I don't think we're willing to pay at the moment.

Comment 25

17 years ago
We have been very careful about performance in this patch and several others which 
lay the foundation for this patch. 
As you might have noticed in my patch, we only added an addtional comparison in 
ResolveForward for some non-surrogate characters. I say some because 
in the fast track code of ResolveForward, this comparison is omitted. So there should
be no noticable performance hit. 
Aren't we going to run into problems in some cases where people use UCS2 to UTF8
conversion, though?  (That will destroy the UTF-16 multi-doublebyte characters,
as well as a few others.)

Comment 27

17 years ago
No, conversion back and forth between UTF8 and UTF16 has been checked in before 
this bug. Yes, to support UTF16 is more than just to display surrogate pair.
This bug is only aimed to fix display problem for the support of UTF16. The 
good thing is, none of those work will bring problem to existing support for 
UCS2 (theoritically of cause). 

Comment 28

17 years ago
Note that the Plane 1 testcases don't appear to work using
FizzillaCFM/2002012503  with the User-Defined font set to Code2001.

Comment 29

17 years ago
I tested this in 04-02 trunk build with Code2001.ttf font.
I saw the problem which Shanjian addressed in comment #11. 
I verified as fixed.

Comment 30

17 years ago
Teruko, how did you get the testcases to display properly? I tried using
FizzillaCFM/2002032915 and set all font classes to Code2001 for User Defined,
but everything in gothic.htm still shows up as question marks.

Comment 31

17 years ago
Greg K.

I used the 04-02 trunk build on Win2k.  After I installed Code2001.ttf font, the
page is displayed correctly.

Comment 32

17 years ago
I forgot to mention that I set the Code2001.ttf font for User-Define language as

1. Open the Preferences dialog
2. Select Appearance -> Fonts, select User-Defined in Font for on the right.
3. In the Proportional, select Serif
4. In the Serif, select Code2001 font

Comment 33

17 years ago
Hm. I did make those font settings.

Since it's not working on Mac OS X, I think I'd better Reopen.
Resolution: FIXED → ---

Comment 34

17 years ago
This bug is for windows only. On windows, there is no need to set font preference. 
Mozilla will pick it up whenever it is available. For other platforms, story might 
be totally different. Implementation depends on OS detail. You can file a new bug 
against MAC. 
Last Resolved: 17 years ago17 years ago
OS: All → Windows 2000
Hardware: All → PC
Resolution: --- → FIXED

Comment 35

17 years ago

Comment 36

17 years ago
I think this bug wasn't Windows-only until you changed it just now. ;)

Honestly, if this bug can be reproduced on multiple platforms (which it can), it
should be All, and shouldn't be resolved until it's fixed on them all. File
platform-specific bugs that block this one, but this one shouldn't be resolved
when it's only been fixed on one platform.

Just my two cents.

Comment 37

17 years ago
This bug was originally filed against windows only. I don't know who changed it 
to all. I just changed it back. 

Surrogate support was managed platform specificly in project level, and that's 
why we don't want another meta bug to do the same thing. I knew there is a bug 
filed against Linux long time ago, but not sure about Mac. I suggest you to file  
one against mac just in case, since I have no idea about the plan and progress 
in surrogate support on Mac. Ftang can give you a detail picture about that. 


15 years ago
Depends on: 232657
You need to log in before you can comment on or make changes to this bug.