102758 - Do not support surrogate correctly in NCR code

Status: VERIFIED WORKSFORME

(Core :: Internationalization, defect)

Description

[Frank Tang]

we use 16 to handle NCR, so we won't be able to encode correctly encoded NCR
between 0x10000 and 0x10ffff

test page http://warp/u/ftang/utf8test/ncr.cgi?p=0x1&r=0x0&c=0x0&face=MS+Sans+Serif 
should not show any ASCII to us, but we do it right now.

Comment 1

[Frank Tang]

yokoyama, can you work on it once shanjian add the surrogate support for WinXP?

Comment 2

[Roy Yokoyama]

accepting and making 102557 as a depend

Comment 3

[Frank Tang]

we need to cover both html and xml. the code to handle xml may differ from the
code handle html.
looks like the code for html is in nsHTMLToken.cpp in mozila/htmlparser/src, not
sure about xml.

Comment 4

[Frank Tang]

Attachment: patch, need testing.

Comment 5

[Frank Tang]

Comment on attachment 64346 [details] [diff] [review]
patch, need testing. 

ignore this , it does not work

Comment 6

[Frank Tang]

forget about my comment. I am not sure where should we change. Someone need to
set a debugger and trace it back from layout.

smontagu, harishd- do you know where so we turn A into 'A' ?

Comment 7

[harishd]

It's done by the expat parser ( mozilla/expat ). Heikki, is that right?

Comment 8

[Heikki Toivonen (remove -bugzilla when emailing directly)]

I believe harishd is right, haven't looked or tested but I can't think of how
else it could be done.

Comment 9

[Roy Yokoyama]

shanjian: your comment in 102557
>No, it isn't working yet. We have problem in parsing NCR value larger than 
>0x10000. I have a fix sitting in my local tree.
Where is at with the fix?  Did it check in?

Comment 10

[Shanjian Li]

Yes, so this bug is resolved. 

Comment 11

[Simon Montagu :smontagu]

On the test page I see the correct characters where my font (Code2001) has them,
but for invalid characters I see ?? instead of ?

Is there an open bug on that problem?

Comment 12

[Shanjian Li]

frank filed one for that problem. 

Comment 13

[Roy Yokoyama]

thanks, shanjian

Comment 14

[Yuying Long]

Verified on 02-11 trunk build on linux and winME, Frank's test page is displayed
as "??" now.

Mark it as verified.