73710 - nsUnicodeDecodeHelper to support overwrapping ranges

Status: RESOLVED FIXED

(Core :: Internationalization, defect)

Description

[nhottanscp]

Currently, overwrapping first byte range as below (0xC6 and 0xF9) is not supported.

60 static uRange g_BIG5HKSCSRanges[] = {
 61 { 0x00, 0x7E },
 62 { 0x81, 0xA0 },
 63 { 0xA1, 0xC6 },
 64 { 0xC6, 0xC8 },
 65 { 0xC9, 0xF9 },
 66 { 0xF9, 0xFE }
 67 };

Comment 1

[Frank Tang]

We need this to fully support HKSCS.

Comment 2

[Frank Tang]

cannot make it. move to moz0.9.1

Comment 3

[Frank Tang]

the problem is in 
144 NS_IMETHODIMP nsUnicodeDecodeHelper::ConvertByMultiTable(

The current logic find a matching range
165     for (i=0; i<aTableCount; i++) 
166       if ((aRangeArray[i].min <= *src) && (*src <= aRangeArray[i].max)) 
break;
167 
and then process line 168-197. If it faild , it won't try the next range. We need 
to change the logic so it will try next range if either uScan reutrn false or 
uMapCode return false. 

nhotta- I am overload. Can you help ?
We need this to fix the remaining HKSCS problems. 

Comment 4

[Frank Tang]

move back from nhotta to ftang and target moz0.9.1
This case both 78614 and 80865

Comment 5

[Frank Tang]

Change this routine should impact the following conversion
EUCKR to Unicode
Big5 to Unicode
BIG5HKSCS to Unicode
EUCTW to Unicode

I may need to undo the part shanjian and                                                                                    
[e-mail] fixed in r1.12 of nsUnicodeDecodeHelper for bug 64235 
and 25037. The fix there seems won't work for some cases. 
[e-mail]- the part of code should not impact ucvcn nor ucvja, but 
it should impact ucvko ucvtw and ucvtw2. Do you have good test cases for EUCKR 
that I can try after I undo that fix?

Comment 6

[Jungshik Shin]

I listed several test cases in comments for bug 64235 and 25037.
( NYTimes, French/German Yahoo, Dell web page, etc). Bug 64235/25037
are about *protecting* mozilla against malformed pages as you may
know.
BTW, it seems like this patch for overlapping range will make it much
easier to implement other encodings to Unicode converters.

Comment 7

[nhottanscp]

Reassign to ftang.

Comment 8

[Frank Tang]

*** Bug 80865 has been marked as a duplicate of this bug. ***

Comment 9

[Frank Tang]

I test all the case in 64235 with my fix. It break nothing. Here is the patch

Comment 10

[Frank Tang]

Attachment: patch

Comment 11

[Frank Tang]

fixed and check in 

Comment 12

[Masaki Katakai]

Ervin, can you check the original problem 80865 is fixed on nightly?

Comment 13

[Ervin Yan]

Verified in nightly 2001051622, bug 80865 is fixed.