25037 - illegal 0xA0 code point in Multibyte charset break parser

Status: VERIFIED FIXED

(Core :: DOM: HTML Parser, defect, P3)

Description

[Teruko Kobayashi]

When you see the above page, the last table on the right displays /TD.

Steps of reproduce
1. Go to above URL
2. Look at the table on the right 
   Under "DAVOS 2000", "/TD" is displayed.

3. Select menu View|Page Source

Look at the Line 

<TD VALIGN=TOP><LI> /TD>
Source does not have "<" before "/TD>"

However, I look at the source of this page in Communicator 4.x.
<TD VALIGN=TOP><LI></TD>

Tested 2000012515 Win32 and Linux build.

Comment 1

[rickg]

This works perfectly for me. Petersen, can you see the problem. Also -- a 
reduced test case would be very helpful.

Comment 2

[bobj]

I reproduced this with the 1/25/2000 M14 build running on Win95.

The strange thing, is that when I saved it to the blues server with the 
intention to try and make a simpler test case, it worked!  I looked at the
saved unmodified file on blues, and it looked the same.  Here is my saved file:
   http://blues/users/bobj/publish/test/zh-tw-index.html

Comment 3

[Chris Petersen]

With the Feb 07, (20000020608), I can't reproduce the problem described.

Comment 4

[rickg]

Harish -- Petersen and I dont see this, but you might give it a try.

Comment 5

[Teruko Kobayashi]

Since the content of http://home.netscape.com/zh/cn/ has been changed, I copied simple page to 
http://jazz/users/teruko/tests/cntest1.html

Comment 6

[harishd]

Not able to reproduce!!!  Teruko, is this bug still valid?

Comment 7

[bobj]

In my 2000-01-26 14:39 comment, I had strange results.  I could reproduce it,
but when I copied the page to another server, it worked...

Comment 8

[harishd]

Which implies that the problem could be server related..right?? 

Comment 9

[Teruko Kobayashi]

Correction of my previous comment.
Since the content of http://home.netscape.com/zh/cn/ has been changed, I copied simple page to 
http://jazz/users/teruko/publish/tests/cntest1.html

harishd,
Did you try http://jazz/users/teruko/publish/tests/cntest1.html?

Comment 10

[Teruko Kobayashi]

Attachment: test case

Comment 11

[harishd]

Attachment: Simpler test case

Comment 12

[harishd]

Could be a bug in ftang's code in nsScanner::Append().  Frank??

Comment 13

[Frank Tang]

Can you reproduce the problem w/ the "Simpler test case" 
http://bugzilla.mozilla.org/showattachment.cgi?attach_id=5105 ?

Comment 14

[Frank Tang]

harishd is right. This is a converter problem. What happen is somehow there are 
a 0xA0 after <LI> . but 0xA0 is not a legal code point in GB2312. somehow our 
error handling code eat two bytes instead of one byte for this and cause the < 
get eaten. Reassign this back to ftang. 

Good catch, harishd. jbetak- is this similar to the one you found in the old 
UTF-8 code ?

Comment 15

[Frank Tang]

I am not sure how many content out there have this kind of illegal code point 
issue. Mark it M18

Comment 16

[[e-mail] (away - not reading bugmail)]

ftang: the problem with UTF-8 before Feb 4 was very similar - we were eating 2 
bytes instead of one. 

My impression was that it was happening in the buffering / error handling code. 
I was not able to discover a similar regularity like in this 0xA0 problem 
though. Will have a look again later this week, maybe they have more in common. 

Referencing to <A HREF="http://bugzilla.mozilla.org/show_bug.cgi?id=8702">Bug 
#8702</A>

Comment 17

[Frank Tang]

Change the summary from  "</TD> parsing problem" to "illegal 0xA0 code point in
Multibyte charset break parser"

In 4.x, we silently support undef 0xa0 code point for CJK multibyte code page,
however, in seamonkey, we don't. This cuase backward compatability issue.
Some page accidentally have this character and the webmaster does not spot the
problem since it is display correctly in 4.x. When SeaMonkey hit it, it could
cause parser problem. This happen especailly if the 0xA0 is before a open tag,
such as <TABLE> , currently SeaMonkey will take the 0xA0 and the next character
to form a undefine character, therefore the '<' of the <TABLE> (or other tag)
will be eat by the converter

We have the following options-
1. Ignore this bug and let the web master fix their page since these character
is not defined in the standard of these charset.
2. Add 0xa0 to the convert to unicode converter for all the multi byte charset
so it will be convert to U+00A0
Mark it M16

Comment 18

[Frank Tang]

*** Bug 27704 has been marked as a duplicate of this bug. ***

Comment 19

[Erik van der Poel]

Mozilla has worked very hard to be compatible with Nav4. This 0xA0 issue should
not be an exception. We should be compatible with Nav4, at least in NavQuirks
mode, and probably even in Strict mode. Even in Strict mode, if there is an A0
byte, we should not eat the next byte, I think. Is this bug serious? Is M16 the
right milestone for this?

Comment 20

[Frank Tang]

converter related issue reassign to cata.

Comment 21

[Frank Tang]

*** Bug 27454 has been marked as a duplicate of this bug. ***

Comment 22

[Frank Tang]

move all cata's bug to ftang

Comment 23

[Frank Tang]

We need to add 0xa0 for Big5, gb2312, EUC-KR, GBK, EUC-JP.
Mark this as moz0.9 P3

Comment 24

[Frank Tang]

shanjian- can you help to take a look at this?

Comment 25

[Shanjian Li]

Attachment: Proposed fix part 1

Comment 26

[Shanjian Li]

Attachment: Proposed fix part 2

Comment 27

[Shanjian Li]

Attachment: Proposed fix part 1

Comment 28

[Shanjian Li]

Attachment: Proposed fix part 4

Comment 29

[Shanjian Li]

In GBK, 0xA0 is a legal lead byte. Patch 3 (wrongly marked as 1) should be 
dropped.

Comment 30

[Shanjian Li]

Attachment: new part2 fix

Comment 31

[Shanjian Li]

When I took a look at bug 64235, I revised fix part2. So problem in 64235 
will be taken care of here.  
For people to review the fix, complete fix include:
fix part 1, (first one, not the one wrongly marked) nsUnicodeDecodeHelper.cpp
new part2 fix, fix GB2312
fix part 4, fix japanese (eucjp and sjis)

Comment 32

[Jungshik Shin]

Bug 64235 is a superset of this bug, IMHO. It has to do with not
just stand-alone 0xA0 but also with any stand-alone byte/octet
with MSB=1 in various CJK encodings. I added a new patch to
take care of it for GBK and GB2312 (to bug 64235)

Comment 33

[bsharma]

updated qa contact.

Comment 34

[Frank Tang]

shanjian- this is hard to review. Why don't you produce a new patch which 
include all the necessary part. 

Comment 35

[Frank Tang]

Is that true the patch in 64235 cover the fix of this ?

Comment 36

[Shanjian Li]

yes, 64235 covered this one.

Comment 37

[Shanjian Li]

fix has been checked. 

Comment 38

[bsharma]

Verified on
build: 2001-06-20-04-Trunk
platform: Win NT

I do not see "/TD".