Closed
Bug 64235
Opened 25 years ago
Closed 25 years ago
mishandling of 'stand-alone' octet with MSB set in CJK(multibyte) encodings
Categories
(Core :: DOM: HTML Parser, defect, P3)
Core
DOM: HTML Parser
Tracking
()
VERIFIED
FIXED
mozilla0.9
People
(Reporter: jshin, Assigned: shanjian)
References
()
Details
(Keywords: intl)
Attachments
(3 files)
|
3.63 KB,
patch
|
Details | Diff | Splinter Review | |
|
2.61 KB,
patch
|
Details | Diff | Splinter Review | |
|
14.19 KB,
patch
|
Details | Diff | Splinter Review |
The page at the URL above has 0xa0 embedded. It gets rendered
all right if character coding is set to ISO-8859-1(or 15),
but in EUC encodings (e.g. EUC-KR, EUC-CN/GB 2312-80,EUC-TW)
Mozilla tries to match up 0xA0 with the following octet
which is "<" (angle-bracket:0x3c) and is an octet with MSB unset.
Because in EUC-KR and EUC-CN there's no character
corresponding to 0xa0 + 0x3c, Mozilla turns the sequence
to a question mark and stop parsing the following
tag (e.g. if '0x80' is followed by <span class="c4">,
Mozilla renders it as a question mark followed
by "span class=....").
It could well be argued that it's the responsibility of
users to set the encoding correctly (or of the author of
the web page
to mark the document in the correct encoding) and this
is not a bug. Even if that's the case, it would be
nice if Mozilla could handle this case more gracefully.
That is, if it detects an octet with MSB set which
stands by itself in EUC-KR and EUC-CN, it should mark
it(but NOT the two octet sequence made of
the octet with MSB set(in the example
above, it's 0xa0) and the following octet
with MSB UNSET) as an unknown character and
roll back by an octet to resume the parsing at
the second octet (with MSB UNSET. in the example
above, it's "<").
Moreover, in this particular instance
0xa0 is NOT valid in any circumstance in EUC-KR
and EUC-CN(and EUC-JP/EUC-TW) and as such Mozilla
should not try to
match it (0xa0) up with the following octet at all. Instead,
it just have to detect it as an unknown character.
P.S. The behavior in EUC-JP is different
| Reporter | ||
Comment 1•25 years ago
|
||
The behavior with character coding set to EUC-JP
is much better than with char. coding set to EUC-KR/CN/TW.
Instead of exposing bare html source after 0x80,
it just renders 0x80 as a question mark
(which is different from the glyph used to mark
a missing glyph in the font - question mark inside
a rhombus).
This sounds more like ftang's area. Frank, could you please take a look? Thanx
Assignee: harishd → ftang
Comment 3•25 years ago
|
||
I think this problem is important enough for us to take a look at this.
Basically, 4.x can handle 0xa0 for nbsp even in some charset which do not
uspport nbsp like EUC-KR, EUC-JP, Shift-JIS . Probably we should at least have a
better fallback instead of treat it as a two byte sequence.
Priority: -- → P3
Target Milestone: --- → mozilla0.9
| Reporter | ||
Comment 5•25 years ago
|
||
This bug is more generic than just not being able to handle 0xa0 (U00A0: NBSP)
in EUC-xx, SJIS,Big5 encodings like I described in my original
report. It applies to the rendering of
any 'stand-alone' octet with MSB set (i.e.any
octet with MSB set that does NOT pair with the following octet
to form a valid two-byte-character ) in two-byte (or multibyte ) encodings
such as EUC-xx, Big5 and SJIS.
For instance,
<http://accessories.us.dell.com/sna/topic.asp?~script=sna&~topic=mem_info>
has '0x99' enclosed within <SUP> and </SUP>. As 0x99 by itself is
not a valid character in none of EUC-xx encodings, '0x99' enclosed between
<SUP> and </SUP> should be rendered as '?' raised to make it a superscript
when the current encoding is EUC-XX. Instead of doing that, Mozilla
interprets '0x99' and the following '<' (with which <SUP tag begins)
as a single entitry (for a two-byte character) and renders it as '?'
and go on to expose the remaning part of <SUP> which 'SUP>'.
Compare what Netscape 4.7x does with two URLs above (one given in
the URL field of the report and the other given inthis comment)
with what Mozilla does with them with both browsers set to use
one of EUC-XX, SJIS, and Big5 encodings. Netscape 4.7x silently
ignores the character in question (thus at least it's aware that
<0x99> or <0xa0> that stands by itself CANNOT be a valid character
in the present encoding) while Mozilla tries to interpret
<0x99> or <0xa0> and the following octet (with MSB unset) and fails.
THen, it goes on to render by doing which it exposes a part
of the "raw html source" such as 'SUP>'
(if the octet following <0x99> or <0xa0> happens to be '<').
I guess this problem has to be dealt with by the encoding converted.
IMHO, fed with the html sources for the two pages mentioned, the converter
should cry out loud that <0x99> and <0xa0> that stands by itself
and followed by '<' is not valid.
Comment 6•25 years ago
|
||
shanjian- can you take a look at this bug ?
Assignee: ftang → shanjian
Status: ASSIGNED → NEW
| Assignee | ||
Comment 7•25 years ago
|
||
this bug should be taken care in 25037. I just revise the fix there.
Status: NEW → ASSIGNED
| Reporter | ||
Comment 8•25 years ago
|
||
The fix for bug 25037 is not extensive enough to cover all the possible
cases of *stand-alone* invalid byte with MSB=1 in various CJK encodings.
There's a flaw with most CJK legacy encoding(EUC-*,SJIS,Big5,GBK) to Unicode
converters in that most of them do NOT check the validity of the second byte
while they do check the validity of the first byte.For instance,
nsGB2312ToUnicode.cpp does NOT check whether the second byte
(contained in 'right' variable around line 100) is valid (it should be
larger than 0xA0 for GB 2312/EUC-CN and larger or equal to 0x40 for
GBK). As a result, even with the patch to fix bug 25037 applied, the following
page (I mentioned before) at
<http://accessories.us.dell.com/sna/topic.asp?~script=sna&~topic=mem_info>
does NOT get rendered "correctly"(that is, ignoring or
marking invalid characters ) when the encoding is set to
SJIS, GB2312, GBK and Big5.
I'm attaching a patch to fix this problem for GBK and GB2312.
I guess fixing this problem for SJIS and Big5 involves patching
intl/uconv/src/nsUnicodeDecoder.cpp.
As for EUC-* which give a false impression
that they don't suffer from this problem, I believe
they do have this problem -
the absence of the second-byte range check-, too. Two pages
(one in the URL field of this report and the other I mentioned before
and I'm referring to again here) do not expose the problem for EUC-*
(after applying the patch for bug 25037 by shanjian) because
the stand-alone byte with MSB=1 in the two pages(0xA0 and 0x99) are NOT valid
as the first byte for EUC-* encodings (so they're filtered and
taken care of even though the second byte is not checked for validity).
I'll make a test page which can expose the problem for EUC-* encodings later
and post here.
| Reporter | ||
Comment 9•25 years ago
|
||
| Reporter | ||
Comment 10•25 years ago
|
||
1. changed the summary line to better describe the problem (EUC encoding -->
CJK multibyte encoding)
2. Unlike other EUC-* encodings, EUC-CN(GB 2312)->Unicode
converter does NOT use shift/map tables so that by patching
nsGB2312ToUnicode.cpp the problem is fixed. For other EUC-*
and SJIS, Big5 encodings, intl/uconv/src/nsUnicodeDecoder.cpp
(and perhaps other files) has to be patched.
.
Summary: mishandling of 'stand-alone' octet with MSB set in EUC encodings → mishandling of 'stand-alone' octet with MSB set in CJK(multibyte) encodings
| Assignee | ||
Comment 11•25 years ago
|
||
JungShik,
I took a look at the patch, it looked fine to me. SJIS and EUC_JP are handled
by nsJapaneseToUnicode.cpp, it does not have this problem. Other encoding like
EUC-Kr, Big5, EUC-TW are handled through nsUnicodeDecoderHelper.cpp, and I do
not see any problem either. Before you present a testcase, I will leave it that
way. Thanks for your help!
| Reporter | ||
Comment 12•25 years ago
|
||
> I took a look at the patch, it looked fine to me. SJIS and EUC_JP
> are handled by nsJapaneseToUnicode.cpp, it does not have this
> problem. Other encoding like EUC-Kr, Big5, EUC-TW are handled through
> nsUnicodeDecoderHelper.cpp, and I do not see any problem either. Before
> you present a testcase, I will leave it that way. Thanks for your help!
Shanjian,
Thank you for taking a look at my patch.
As for SJIS and EUC-JP, you may wish to view the following two pages
<http://accessories.us.dell.com/sna/topic.asp?~script=sna&~topic=mem_info>
http://adc.gsfc.nasa.gov
The former has 0x99 embedded in otherwise pure US-ASCII page. With the
encoding set to SJIS and Big5, raw HTML tags are exposed (The second with
0xA0 has the same problem for Big5 while it doesn't for SJIS) because the
second byte range check is not in place and 0x99 is a valid lead byte for
SJIS and Big5. On the other hand, with my patch (on top of your patch)
applied and with the encoding set to GB 2312(EUC-CN) and GBK, Mozilla
just marks 0x99 as '?' and does NOT eat up the following byte because my
patch adds the range check for the second byte. This range check does
not work always for GBK because its second byte range overlaps with
US-ASCII while the second byte of EUC-CN(GB 2312) does NOT That is,
in EUC-CN/GB 2312 two coded character set US-ASCII and GB 2312-80 have
cleanly separated code spaces wheras that's not the case in GBK, SJIS,
UHC (Windows-949) and JOHAB. Anyway, my patch for GBK does as much as
it can to guard Mozilla against 'stray' invalid byte with MSB=1.
A similar patch for the second byte range check(although it can never
be perfect due to the aforementioned overlap of the code space) has to
be made for SJIS, Big5 (and to-be-implemented UHC and JOHAB).
The problem with EUC-KR, EUC-JP and EUC-TW doesn't reveal itself in
the page above. However, if you point Mozilla to French and German pages
(for instance, <http://de.yahoo.com>, <http://fr.yahoo.com>) with
a lot of umlauts, accents,grave accent, etc with the encoding set to
EUC-{JP,TW,KR}, you'll see what I mean (It turns out that I don't have to
make a test case) Even NYTimes has some accented characters mixed in from
time to time and I believe it's a good idea to render a stand-alone byte
with MSB=1 as '?' and go on to parse next byte (which is very likely to be
just a plain US-ASCII char) with the encoding set to one of CJK multibyte
encodings instead of eating up the next byte and rendering the byte in
question and the next byte together as "incongruous" CJK characters.
Some may question why Mozilla has to take care of this rather than
just asking users to switch the encoding (or better ask the authors/web
masters of all those ISO-8859-1 pages without MIME charset info. to add
that info. to their pages). In the ideal world, that's a perfectly good
solution. However, it would not happen overnight (I've written to the
webmasters of NYTimes and other newspapers several times suggesting that
they put MIME charset info. on their webpages, but have yet to hear a
single response). In the meantime, CJK users have to switch the encoding
back and forth (auto-detect works pretty well, but still...)
Incidentally, you may just leave nsUnicodeDecoderHelper.cpp as it is now.
I think something similar to what's done for GB 2312 and GBK (a rewrite
of the encoding converters which don't rely on shiftable/maps) might
have to be done for Korean (EUC-KR, Windows-949/CP949) and traditional
Chinese encodings(Big5, EUC-TW).
I'm attaching a patch to nsJapaneseToUnicode.cpp which makes
http://accessories.us.dell.com/sna/topic.asp?~script=sna&~topic=mem_info
rendered better with the encoding set to SJIS. I also changed the part
for EUC-JP in the same file, but didn't change the part dealing with
JIS X 0212 and JIS X 0201 because I'm not sure what's the best way to
deal with invalid EUC-JP seq. for JIS X 0212 (0x8f followed by (an)
invalid bytes or JIS X 0201 (0x8e followed by an invalid byte
Jungshik
| Reporter | ||
Comment 13•25 years ago
|
||
| Assignee | ||
Comment 14•25 years ago
|
||
Jungshik,
Now I understand your intention and argument better. We can take care of
the remaining issues in nsUnicodeDecodeHelper.cpp. Around line 130, this should
work:
+ if (bcr == 2 && !(*(src+1) & 0x80) )
+ {
+ bcr--;
+ }
// Unicode replacement value for unmappable chars
*dest = 0xfffd;
Since you have all test cases ready and know what to look for, can you try it and finalize the patch?
Thanks a lot!
| Reporter | ||
Comment 15•25 years ago
|
||
Shanjian,
Thank you for leading me into the right place. I was looking there,
but I was wrestling with the details of the implementation of the
converters (uscan(), umapCode(), etc)
> Now I understand your intention and argument better. We can take care of
Glad that I was more successful in getting myself understood :-). BTW,
more briefly put, I want pages like <http://de.yahoo.com>(in ISO-8859-1)
to be rendered more or less the same way whether US-ASCII is chosen or
various CJK encodings are chosen.
> the remaining issues in nsUnicodeDecodeHelper.cpp. Around line 130, this
> should work:
> + if (bcr == 2 && !(*(src+1) & 0x80) )
> + {
> + bcr--;
> + }
> // Unicode replacement value for unmappable chars
> *dest = 0xfffd;
I guess this should be put about 20 lines later than line 130. Line
130 is for ConvertByTable() and EUC-TW,Big5,and EUC-KR (indirectly)
call ConvertbyMultiTable(). Indeed, adding it to ConvertByTable() didn't
make any change. It was only after it's added to ConvertByMultitable()
that <http://de.yahoo.com> and other two pages mentioned before (in
almost pure US-ASCII with one or two non-US-ASCII characters sneaked in)
began to get rendered (with the encoding set to Big5) as I wished.
However, a strange thing is happening with EUC-KR and EUC-TW.
Let me explain what I observed with EUC-KR first (which is simpler than
EUC-TW). If a byte with MSB=1 (> 0xa0) is followed by a byte (<= 0x20),
it works fine. That is, a stand-alone byte with MSB=1 (> 0xa0) is rendered
as "?" as intended and the following byte (e.g. space:0x20 or lf:0x0a) is
fed back to the parser instead of being eaten up along with the preceeding
stand-alone byte with MSB=1. To my surprise if a stand-alone byte with
MSB=1 (let me refer to it as b1 > 0xa0: the case of stand-alone byte
with MSB=1 < 0xa0 is taken care of elsewhere - by range-check for the
first byte- for EUC-* encodings) is followed by a byte with MSB=0(let me
denote it by b2 >0x20 ), two byte-sequence of b1 and b2 is considered as
equivalent to the sequence of b1 followed by b2 & 0x80 and rendered as
such. For instance, <http://de.yahoo.com> has several sequence of 0xb7
and 0x3c ( "<" : less than ) in the section "Aktuell". Those sequences
are rendered as 0xb7 and 0xbc (=0x3c & 0x80) which is HANGUL Syallable
RIEUL-E-RIEUL (UB810). To make sure, I inserted the following right before
'if (!uMapCode((uTable*) aMappingTable[i]....' in ConvertByMultitable()
if ( (*src & 0x80) && ! (*(src+1) & 0x80) )
{
fprintf(stderr,"JS(3): bcr=%d src=%c:%2x;(src+1)=%c:%2x\n",
bcr,*src,(PRUint16)*src,*(src+1),(PRUint16)*(src+1));
}
Inside the 'if'-clause, I also inserted a similar diagnostic output
before 'if (bcr == 2 && !(*(src+1) & 0x80) ) {...'
fprintf(stderr,"JS(4): bcr=%d src=%c:%2x;(src+1)=%c:%2x\n",
bcr,*src,(PRUint16)*src,*(src+1),(PRUint16)*(src+1));
As I suspected, for the sequence of 0xb7 and 0x3c I got the output
from only the first line
JS(3): bcr=2 src=b7;(src+1)=<:3c
while for the sequence of 0xb7 and 0x20 both lines got reached to produceJS(3):
bcr=2 src=b7;(src+1)= :20
JS(4): bcr=2 src=b7;(src+1)= :20
(please, note that I'm omitting the actual character output for 0xb7
which is the middle dot in ISO-8859-1).
So, what's the cause of this? As mentioned above, it's uMapCode() that's
to blame. For EUC-KR 'proper' (that is different from Windows-949
which is used for MS-Windows version when MS_EUCKR is defined),
g_EUCKRShiftTable[] has the classID of u2BytesGRCharset corresponding to
the 4th element of m_scanner[] array in intl/uconv/src/uscan.c which is
uCheckAndScanAlways2ByteShiftGR(). This function, contrary to what its
name implies, does not check thoroughly enough for a 94x94(or 96x96)
character set invoked on GR. It checks the input buffer length but
doesn't check whether both bytes in the input buffer are in the valid
range ( [0xa1,0xfe] for 94x94 and [0xa0,0xff] for 96x96). Consequently,
2-byte sequences like 0xb7 and 0x3c pass thru and return PR_TRUE. To
filter them out, I added this check to uCheckAndScanAlways2ByteShiftGR()
and other related functions for EUC-* encodings.
// Both bytes should be in the range of [0xa1,0xfe] for 94x94 character sets
// invoked on GR. No encoding implemented in Mozilla uses 96x96 char. sets.
if(inbuflen < 2 || ! CHK_GR94_2Byte(in[1],in[0]))
return PR_FALSE;
else
{
*inscanlen = 2;
*out = (((in[0] << 8) | ( in[1])) & 0x7F7F);
return PR_TRUE;
}
along with two macro definitions as below
#define CHK_GR94(b) ( (PRUint8) 0xa0 < (PRUint8) (b) && \
(PRUint8) (b) < (PRUint8) 0xff )
#define CHK_GR94_2Byte(b1,b2) (CHK_GR94(b1) && CHK_GR94(b2))
With this guard in place, all the test cases gave me what I expected
them to. <http://adc.gsfc.nasa.gov> has 4 consequtive 0xa0's which
cannot help being interpreted as two valid 2-byte characters
in Big5 and GBK (because a pair of 0xa0's are valid in those
encodings). The same is true of some two byte sequences of MSB=1
byte followed by MSB=0 byte in ISO-8859-1 pages when viewed
with the encoding set to Big5, GBK and Shift-JIS. For instance,
<http://accessories.us.dell.com/sna/topic.asp?~script=sna&~topic=mem_info>
has 0xa0 followed by 0x50. 0xa0-0x50 pair is valid in GBK (U71E9) and
rendered as U71E9(I couldn't confirm this on Linux because I couldn't
find GBK fonts for X11. I might have 'borrowed' Windows ttf font for
CP936 for a while)
| Reporter | ||
Comment 16•25 years ago
|
||
| Assignee | ||
Comment 17•25 years ago
|
||
Agreed, I will get sr and check it in.
Comment 18•25 years ago
|
||
Have you test your latest patch with Big5? I am worry that the GR94 macro may
hurt Big5, GBK, Shift_JIS and CP949 (if somehow we have error in the past)
| Reporter | ||
Comment 19•25 years ago
|
||
My patch which adds the validity check for GR 94 character
sets should not affect non-ISO-2022 compliant encodings like Big5,
CP949, Shift_JIS, GBK. (the latter two are handled separately anyway
by nsJapanese...cpp and nsGBKToUnicode.cpp). That's because Big5
shiftable has the classID of u2BytesCharset which leads to the
invocation of uCheckAndScanAlways2Byte in uscan.c which I haven't
touched. EUC-KR and EUC-TW have the classID of u2BytesGRCharset so that
uCheckAndScanAlways2ByteShiftGR (which I modified) is used.
Hmm, this I guess you already know. Am I missing something? I tested
Big5 pages with my latest patch applied (not extensively though) and I
couldn't find any obvious problem. (I can't test it right now because
I'm too far away from my Linux box in terms of net distance to run
Mozilla remotely). I'll try some more test later today.
Jungshik
Comment 21•25 years ago
|
||
r=ftang
| Reporter | ||
Comment 22•25 years ago
|
||
I've tried various pages in Big5 starting at <http://tw.yahoo.com/> and
couldn't find any anomaly. As far as I can tell, the patch seems to work
fine.
| Reporter | ||
Comment 23•25 years ago
|
||
Yet another example to demonstrate why this patch is necessary :-)
Point mozilla to <http://www.nytimes.com/pages/opinion/editorial/>
with the encoding set to any of CJK multi byte encodings.
There's 0x95 before the titles of three editorials listed in the page
which gets Mozilla to eat out the following 'less than' and to exposes "a
href=......". With the patch applied, 0x95 is just rendered as '?' minimizing
the 'damage'.
| Assignee | ||
Comment 24•25 years ago
|
||
erik, can you do a sr=?
Comment 25•25 years ago
|
||
sr=erik
| Assignee | ||
Comment 26•25 years ago
|
||
checked in the fix.
Status: ASSIGNED → RESOLVED
Closed: 25 years ago
Resolution: --- → FIXED
Comment 27•25 years ago
|
||
Verified on:
build: 2001-06-04-09-Mtrunk
platform: WinNT
Marking it verified as per above developer comments.
Status: RESOLVED → VERIFIED
You need to log in
before you can comment on or make changes to this bug.
Description
•