Closed Bug 64235 Opened 25 years ago Closed 25 years ago

mishandling of 'stand-alone' octet with MSB set in CJK(multibyte) encodings

Categories

(Core :: DOM: HTML Parser, defect, P3)

defect

Tracking

()

VERIFIED FIXED
mozilla0.9

People

(Reporter: jshin, Assigned: shanjian)

References

()

Details

(Keywords: intl)

Attachments

(3 files)

The page at the URL above has 0xa0 embedded. It gets rendered all right if character coding is set to ISO-8859-1(or 15), but in EUC encodings (e.g. EUC-KR, EUC-CN/GB 2312-80,EUC-TW) Mozilla tries to match up 0xA0 with the following octet which is "<" (angle-bracket:0x3c) and is an octet with MSB unset. Because in EUC-KR and EUC-CN there's no character corresponding to 0xa0 + 0x3c, Mozilla turns the sequence to a question mark and stop parsing the following tag (e.g. if '0x80' is followed by <span class="c4">, Mozilla renders it as a question mark followed by "span class=...."). It could well be argued that it's the responsibility of users to set the encoding correctly (or of the author of the web page to mark the document in the correct encoding) and this is not a bug. Even if that's the case, it would be nice if Mozilla could handle this case more gracefully. That is, if it detects an octet with MSB set which stands by itself in EUC-KR and EUC-CN, it should mark it(but NOT the two octet sequence made of the octet with MSB set(in the example above, it's 0xa0) and the following octet with MSB UNSET) as an unknown character and roll back by an octet to resume the parsing at the second octet (with MSB UNSET. in the example above, it's "<"). Moreover, in this particular instance 0xa0 is NOT valid in any circumstance in EUC-KR and EUC-CN(and EUC-JP/EUC-TW) and as such Mozilla should not try to match it (0xa0) up with the following octet at all. Instead, it just have to detect it as an unknown character. P.S. The behavior in EUC-JP is different
Keywords: intl
The behavior with character coding set to EUC-JP is much better than with char. coding set to EUC-KR/CN/TW. Instead of exposing bare html source after 0x80, it just renders 0x80 as a question mark (which is different from the glyph used to mark a missing glyph in the font - question mark inside a rhombus).
This sounds more like ftang's area. Frank, could you please take a look? Thanx
Assignee: harishd → ftang
I think this problem is important enough for us to take a look at this. Basically, 4.x can handle 0xa0 for nbsp even in some charset which do not uspport nbsp like EUC-KR, EUC-JP, Shift-JIS . Probably we should at least have a better fallback instead of treat it as a two byte sequence.
Priority: -- → P3
Target Milestone: --- → mozilla0.9
mark it as nsbeta1 and ASSIGNED
Status: NEW → ASSIGNED
Keywords: nsbeta1
This bug is more generic than just not being able to handle 0xa0 (U00A0: NBSP) in EUC-xx, SJIS,Big5 encodings like I described in my original report. It applies to the rendering of any 'stand-alone' octet with MSB set (i.e.any octet with MSB set that does NOT pair with the following octet to form a valid two-byte-character ) in two-byte (or multibyte ) encodings such as EUC-xx, Big5 and SJIS. For instance, <http://accessories.us.dell.com/sna/topic.asp?~script=sna&~topic=mem_info> has '0x99' enclosed within <SUP> and </SUP>. As 0x99 by itself is not a valid character in none of EUC-xx encodings, '0x99' enclosed between <SUP> and </SUP> should be rendered as '?' raised to make it a superscript when the current encoding is EUC-XX. Instead of doing that, Mozilla interprets '0x99' and the following '<' (with which <SUP tag begins) as a single entitry (for a two-byte character) and renders it as '?' and go on to expose the remaning part of <SUP> which 'SUP>'. Compare what Netscape 4.7x does with two URLs above (one given in the URL field of the report and the other given inthis comment) with what Mozilla does with them with both browsers set to use one of EUC-XX, SJIS, and Big5 encodings. Netscape 4.7x silently ignores the character in question (thus at least it's aware that <0x99> or <0xa0> that stands by itself CANNOT be a valid character in the present encoding) while Mozilla tries to interpret <0x99> or <0xa0> and the following octet (with MSB unset) and fails. THen, it goes on to render by doing which it exposes a part of the "raw html source" such as 'SUP>' (if the octet following <0x99> or <0xa0> happens to be '<'). I guess this problem has to be dealt with by the encoding converted. IMHO, fed with the html sources for the two pages mentioned, the converter should cry out loud that <0x99> and <0xa0> that stands by itself and followed by '<' is not valid.
shanjian- can you take a look at this bug ?
Assignee: ftang → shanjian
Status: ASSIGNED → NEW
this bug should be taken care in 25037. I just revise the fix there.
Status: NEW → ASSIGNED
The fix for bug 25037 is not extensive enough to cover all the possible cases of *stand-alone* invalid byte with MSB=1 in various CJK encodings. There's a flaw with most CJK legacy encoding(EUC-*,SJIS,Big5,GBK) to Unicode converters in that most of them do NOT check the validity of the second byte while they do check the validity of the first byte.For instance, nsGB2312ToUnicode.cpp does NOT check whether the second byte (contained in 'right' variable around line 100) is valid (it should be larger than 0xA0 for GB 2312/EUC-CN and larger or equal to 0x40 for GBK). As a result, even with the patch to fix bug 25037 applied, the following page (I mentioned before) at <http://accessories.us.dell.com/sna/topic.asp?~script=sna&~topic=mem_info> does NOT get rendered "correctly"(that is, ignoring or marking invalid characters ) when the encoding is set to SJIS, GB2312, GBK and Big5. I'm attaching a patch to fix this problem for GBK and GB2312. I guess fixing this problem for SJIS and Big5 involves patching intl/uconv/src/nsUnicodeDecoder.cpp. As for EUC-* which give a false impression that they don't suffer from this problem, I believe they do have this problem - the absence of the second-byte range check-, too. Two pages (one in the URL field of this report and the other I mentioned before and I'm referring to again here) do not expose the problem for EUC-* (after applying the patch for bug 25037 by shanjian) because the stand-alone byte with MSB=1 in the two pages(0xA0 and 0x99) are NOT valid as the first byte for EUC-* encodings (so they're filtered and taken care of even though the second byte is not checked for validity). I'll make a test page which can expose the problem for EUC-* encodings later and post here.
1. changed the summary line to better describe the problem (EUC encoding --> CJK multibyte encoding) 2. Unlike other EUC-* encodings, EUC-CN(GB 2312)->Unicode converter does NOT use shift/map tables so that by patching nsGB2312ToUnicode.cpp the problem is fixed. For other EUC-* and SJIS, Big5 encodings, intl/uconv/src/nsUnicodeDecoder.cpp (and perhaps other files) has to be patched. .
Summary: mishandling of 'stand-alone' octet with MSB set in EUC encodings → mishandling of 'stand-alone' octet with MSB set in CJK(multibyte) encodings
JungShik, I took a look at the patch, it looked fine to me. SJIS and EUC_JP are handled by nsJapaneseToUnicode.cpp, it does not have this problem. Other encoding like EUC-Kr, Big5, EUC-TW are handled through nsUnicodeDecoderHelper.cpp, and I do not see any problem either. Before you present a testcase, I will leave it that way. Thanks for your help!
> I took a look at the patch, it looked fine to me. SJIS and EUC_JP > are handled by nsJapaneseToUnicode.cpp, it does not have this > problem. Other encoding like EUC-Kr, Big5, EUC-TW are handled through > nsUnicodeDecoderHelper.cpp, and I do not see any problem either. Before > you present a testcase, I will leave it that way. Thanks for your help! Shanjian, Thank you for taking a look at my patch. As for SJIS and EUC-JP, you may wish to view the following two pages <http://accessories.us.dell.com/sna/topic.asp?~script=sna&~topic=mem_info> http://adc.gsfc.nasa.gov The former has 0x99 embedded in otherwise pure US-ASCII page. With the encoding set to SJIS and Big5, raw HTML tags are exposed (The second with 0xA0 has the same problem for Big5 while it doesn't for SJIS) because the second byte range check is not in place and 0x99 is a valid lead byte for SJIS and Big5. On the other hand, with my patch (on top of your patch) applied and with the encoding set to GB 2312(EUC-CN) and GBK, Mozilla just marks 0x99 as '?' and does NOT eat up the following byte because my patch adds the range check for the second byte. This range check does not work always for GBK because its second byte range overlaps with US-ASCII while the second byte of EUC-CN(GB 2312) does NOT That is, in EUC-CN/GB 2312 two coded character set US-ASCII and GB 2312-80 have cleanly separated code spaces wheras that's not the case in GBK, SJIS, UHC (Windows-949) and JOHAB. Anyway, my patch for GBK does as much as it can to guard Mozilla against 'stray' invalid byte with MSB=1. A similar patch for the second byte range check(although it can never be perfect due to the aforementioned overlap of the code space) has to be made for SJIS, Big5 (and to-be-implemented UHC and JOHAB). The problem with EUC-KR, EUC-JP and EUC-TW doesn't reveal itself in the page above. However, if you point Mozilla to French and German pages (for instance, <http://de.yahoo.com>, <http://fr.yahoo.com>) with a lot of umlauts, accents,grave accent, etc with the encoding set to EUC-{JP,TW,KR}, you'll see what I mean (It turns out that I don't have to make a test case) Even NYTimes has some accented characters mixed in from time to time and I believe it's a good idea to render a stand-alone byte with MSB=1 as '?' and go on to parse next byte (which is very likely to be just a plain US-ASCII char) with the encoding set to one of CJK multibyte encodings instead of eating up the next byte and rendering the byte in question and the next byte together as "incongruous" CJK characters. Some may question why Mozilla has to take care of this rather than just asking users to switch the encoding (or better ask the authors/web masters of all those ISO-8859-1 pages without MIME charset info. to add that info. to their pages). In the ideal world, that's a perfectly good solution. However, it would not happen overnight (I've written to the webmasters of NYTimes and other newspapers several times suggesting that they put MIME charset info. on their webpages, but have yet to hear a single response). In the meantime, CJK users have to switch the encoding back and forth (auto-detect works pretty well, but still...) Incidentally, you may just leave nsUnicodeDecoderHelper.cpp as it is now. I think something similar to what's done for GB 2312 and GBK (a rewrite of the encoding converters which don't rely on shiftable/maps) might have to be done for Korean (EUC-KR, Windows-949/CP949) and traditional Chinese encodings(Big5, EUC-TW). I'm attaching a patch to nsJapaneseToUnicode.cpp which makes http://accessories.us.dell.com/sna/topic.asp?~script=sna&~topic=mem_info rendered better with the encoding set to SJIS. I also changed the part for EUC-JP in the same file, but didn't change the part dealing with JIS X 0212 and JIS X 0201 because I'm not sure what's the best way to deal with invalid EUC-JP seq. for JIS X 0212 (0x8f followed by (an) invalid bytes or JIS X 0201 (0x8e followed by an invalid byte Jungshik
Jungshik, Now I understand your intention and argument better. We can take care of the remaining issues in nsUnicodeDecodeHelper.cpp. Around line 130, this should work: + if (bcr == 2 && !(*(src+1) & 0x80) ) + { + bcr--; + } // Unicode replacement value for unmappable chars *dest = 0xfffd; Since you have all test cases ready and know what to look for, can you try it and finalize the patch? Thanks a lot!
Shanjian, Thank you for leading me into the right place. I was looking there, but I was wrestling with the details of the implementation of the converters (uscan(), umapCode(), etc) > Now I understand your intention and argument better. We can take care of Glad that I was more successful in getting myself understood :-). BTW, more briefly put, I want pages like <http://de.yahoo.com>(in ISO-8859-1) to be rendered more or less the same way whether US-ASCII is chosen or various CJK encodings are chosen. > the remaining issues in nsUnicodeDecodeHelper.cpp. Around line 130, this > should work: > + if (bcr == 2 && !(*(src+1) & 0x80) ) > + { > + bcr--; > + } > // Unicode replacement value for unmappable chars > *dest = 0xfffd; I guess this should be put about 20 lines later than line 130. Line 130 is for ConvertByTable() and EUC-TW,Big5,and EUC-KR (indirectly) call ConvertbyMultiTable(). Indeed, adding it to ConvertByTable() didn't make any change. It was only after it's added to ConvertByMultitable() that <http://de.yahoo.com> and other two pages mentioned before (in almost pure US-ASCII with one or two non-US-ASCII characters sneaked in) began to get rendered (with the encoding set to Big5) as I wished. However, a strange thing is happening with EUC-KR and EUC-TW. Let me explain what I observed with EUC-KR first (which is simpler than EUC-TW). If a byte with MSB=1 (> 0xa0) is followed by a byte (<= 0x20), it works fine. That is, a stand-alone byte with MSB=1 (> 0xa0) is rendered as "?" as intended and the following byte (e.g. space:0x20 or lf:0x0a) is fed back to the parser instead of being eaten up along with the preceeding stand-alone byte with MSB=1. To my surprise if a stand-alone byte with MSB=1 (let me refer to it as b1 > 0xa0: the case of stand-alone byte with MSB=1 < 0xa0 is taken care of elsewhere - by range-check for the first byte- for EUC-* encodings) is followed by a byte with MSB=0(let me denote it by b2 >0x20 ), two byte-sequence of b1 and b2 is considered as equivalent to the sequence of b1 followed by b2 & 0x80 and rendered as such. For instance, <http://de.yahoo.com> has several sequence of 0xb7 and 0x3c ( "<" : less than ) in the section "Aktuell". Those sequences are rendered as 0xb7 and 0xbc (=0x3c & 0x80) which is HANGUL Syallable RIEUL-E-RIEUL (UB810). To make sure, I inserted the following right before 'if (!uMapCode((uTable*) aMappingTable[i]....' in ConvertByMultitable() if ( (*src & 0x80) && ! (*(src+1) & 0x80) ) { fprintf(stderr,"JS(3): bcr=%d src=%c:%2x;(src+1)=%c:%2x\n", bcr,*src,(PRUint16)*src,*(src+1),(PRUint16)*(src+1)); } Inside the 'if'-clause, I also inserted a similar diagnostic output before 'if (bcr == 2 && !(*(src+1) & 0x80) ) {...' fprintf(stderr,"JS(4): bcr=%d src=%c:%2x;(src+1)=%c:%2x\n", bcr,*src,(PRUint16)*src,*(src+1),(PRUint16)*(src+1)); As I suspected, for the sequence of 0xb7 and 0x3c I got the output from only the first line JS(3): bcr=2 src=b7;(src+1)=<:3c while for the sequence of 0xb7 and 0x20 both lines got reached to produceJS(3): bcr=2 src=b7;(src+1)= :20 JS(4): bcr=2 src=b7;(src+1)= :20 (please, note that I'm omitting the actual character output for 0xb7 which is the middle dot in ISO-8859-1). So, what's the cause of this? As mentioned above, it's uMapCode() that's to blame. For EUC-KR 'proper' (that is different from Windows-949 which is used for MS-Windows version when MS_EUCKR is defined), g_EUCKRShiftTable[] has the classID of u2BytesGRCharset corresponding to the 4th element of m_scanner[] array in intl/uconv/src/uscan.c which is uCheckAndScanAlways2ByteShiftGR(). This function, contrary to what its name implies, does not check thoroughly enough for a 94x94(or 96x96) character set invoked on GR. It checks the input buffer length but doesn't check whether both bytes in the input buffer are in the valid range ( [0xa1,0xfe] for 94x94 and [0xa0,0xff] for 96x96). Consequently, 2-byte sequences like 0xb7 and 0x3c pass thru and return PR_TRUE. To filter them out, I added this check to uCheckAndScanAlways2ByteShiftGR() and other related functions for EUC-* encodings. // Both bytes should be in the range of [0xa1,0xfe] for 94x94 character sets // invoked on GR. No encoding implemented in Mozilla uses 96x96 char. sets. if(inbuflen < 2 || ! CHK_GR94_2Byte(in[1],in[0])) return PR_FALSE; else { *inscanlen = 2; *out = (((in[0] << 8) | ( in[1])) & 0x7F7F); return PR_TRUE; } along with two macro definitions as below #define CHK_GR94(b) ( (PRUint8) 0xa0 < (PRUint8) (b) && \ (PRUint8) (b) < (PRUint8) 0xff ) #define CHK_GR94_2Byte(b1,b2) (CHK_GR94(b1) && CHK_GR94(b2)) With this guard in place, all the test cases gave me what I expected them to. <http://adc.gsfc.nasa.gov> has 4 consequtive 0xa0's which cannot help being interpreted as two valid 2-byte characters in Big5 and GBK (because a pair of 0xa0's are valid in those encodings). The same is true of some two byte sequences of MSB=1 byte followed by MSB=0 byte in ISO-8859-1 pages when viewed with the encoding set to Big5, GBK and Shift-JIS. For instance, <http://accessories.us.dell.com/sna/topic.asp?~script=sna&~topic=mem_info> has 0xa0 followed by 0x50. 0xa0-0x50 pair is valid in GBK (U71E9) and rendered as U71E9(I couldn't confirm this on Linux because I couldn't find GBK fonts for X11. I might have 'borrowed' Windows ttf font for CP936 for a while)
Agreed, I will get sr and check it in.
Have you test your latest patch with Big5? I am worry that the GR94 macro may hurt Big5, GBK, Shift_JIS and CP949 (if somehow we have error in the past)
My patch which adds the validity check for GR 94 character sets should not affect non-ISO-2022 compliant encodings like Big5, CP949, Shift_JIS, GBK. (the latter two are handled separately anyway by nsJapanese...cpp and nsGBKToUnicode.cpp). That's because Big5 shiftable has the classID of u2BytesCharset which leads to the invocation of uCheckAndScanAlways2Byte in uscan.c which I haven't touched. EUC-KR and EUC-TW have the classID of u2BytesGRCharset so that uCheckAndScanAlways2ByteShiftGR (which I modified) is used. Hmm, this I guess you already know. Am I missing something? I tested Big5 pages with my latest patch applied (not extensively though) and I couldn't find any obvious problem. (I can't test it right now because I'm too far away from my Linux box in terms of net distance to run Mozilla remotely). I'll try some more test later today. Jungshik
updated qa contact.
QA Contact: janc → bsharma
r=ftang
I've tried various pages in Big5 starting at <http://tw.yahoo.com/> and couldn't find any anomaly. As far as I can tell, the patch seems to work fine.
Yet another example to demonstrate why this patch is necessary :-) Point mozilla to <http://www.nytimes.com/pages/opinion/editorial/> with the encoding set to any of CJK multi byte encodings. There's 0x95 before the titles of three editorials listed in the page which gets Mozilla to eat out the following 'less than' and to exposes "a href=......". With the patch applied, 0x95 is just rendered as '?' minimizing the 'damage'.
erik, can you do a sr=?
sr=erik
checked in the fix.
Status: ASSIGNED → RESOLVED
Closed: 25 years ago
Resolution: --- → FIXED
Verified on: build: 2001-06-04-09-Mtrunk platform: WinNT Marking it verified as per above developer comments.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: