116882 - A middle dot character is not displayed on this page

Status: VERIFIED FIXED

(Core :: Internationalization, defect, P3)

Description

[Katsuhiko Momoi]

** Observed with 2001-12-22 Win32 trunk build **

On the above page, there is one character which is not displayed
properly with Mozilla under Shift_JIS encoding. 

It looks like the character has the codepoint 0x81.

(There is a similar bug filed -- Bug 116880. But in that bug the codepoint for
  the problem character is 0x86 0xA6.) 

Neither NN4 nor IE 5.5. has a propblem in displaying this character.

Comment 1

[Roy Yokoyama]

over to Mr.Li.  

Comment 2

[Shanjian Li]

The character in question is 0x81, which is followed by 0x20. 0x8120 is not a 
legal sjis byte sequence. It is very strange to see that both IE and Netscape4.x 
replace such sequence to 0x8145, which is middle-dot. But anyway, I don't think 
this is a mozilla problem. I believe mozilla's behavior is better than both IE 
and Netscape4.x. Why replace illegal byte sequence to 0x8145? (I tried another 
byte sequence 0x8136, which was also replaced by 0x8145.)

Comment 3

[Frank Tang]

sorry, I cannot tell which character you refere to.

Comment 4

[Katsuhiko Momoi]

> I believe mozilla's behavior is better than both IE 
> and Netscape4.x. Why replace illegal byte sequence to 0x8145?

Windows applications when they use Windows OS converters
map this codepoint to the middle dot character. I am sorry
but this is expected on Windows. The character is apparently
fairly widely used -- right or wrong. If you use Notepad, 
Word, and other Windows applications, you see the same
character, not "not found" character as we do on Mozilla.

How are we going to convince Windows users that what they
see in every other application is wrong? 

Let me re-open this for re-consideration and let me provide
additional facts.

ftang: If you want to see which character we are referring to,
just open the URL with Mozilla and compare it with NN4 or IE5/6.
You will see one character with a question mark with Mozilla
but expressed with a middle-dot character in other browsers
and applications.

Comment 5

[Shanjian Li]

Kat, I am not convinced yet. Is this kind of practice common? Did user 
do this intentionally? I mean when they put 0x81, what they want is mid dot?
If MS just take 0x81 and map it to mid dot, that will be easy to understand 
it as a "feature". But to map a range of code points to one character does not 
make much sense.  Can you tell me how such page is created? 

Comment 6

[Frank Tang]

momoi, please attach a screen shot here (and circle with mark) . I cannot see
that ? mark. 

Comment 7

[Katsuhiko Momoi]

In comment 5, shanjian said:

> Is this kind of practice common? Did user do this intentionally? 
...
>Can you tell me how such page is created? 

Yes, this is the question we should be asking before we
decide on this bug. Let me dig around a bit more before
making a decision one way or the other. I suspect this is
an intentional character.

Comment 8

[Katsuhiko Momoi]

Attachment: This is an image which points to the problem character. Compare this to NN4 or IE 5/6

Comment 9

[Frank Tang]

let's try to fix SJIS to Unicode conversion to map 0x8120 to U+30fb so we have
backward compatability ?
reassign back to ftang and mark it as M1.0

Comment 10

[Shanjian Li]

As I mentioned in my previous comment, at least 0x8120 and 0x8136 are mapped to 
u30fb. I believe all characters in 0x8120 to 0x813f are mapped to u30fb, probably
even larger. Adding such nonsense conversion just for this page does not make any 
sense, unless momoi's investigation show that this is a common practice and many 
webpages are doing it. In our charset detector, 0x8120 to 0x813f are illegal byte
sequence. That may confuse some users when they switch detector on and off.

Comment 11

[nhottanscp]

nsbeta1+ per i18n triage

Comment 12

[Frank Tang]

let's fix this.

Comment 13

[Frank Tang]

p3

Comment 14

[Frank Tang]

move to m0.9.9

Comment 15

[Frank Tang]

let's merge this bug into 116882. basically , we want compatible with IE6 on
error handling to reduce risk of site compatability.

What I found by looking at IE6 is the following
a. IE6 treat 0xfd - 0xff as single byte. and convert them into f8f1-f8f3. We
currently treat it as 2 bytes characters and convert to fffd
b. if a lead byte is legal shift jis range but the 2nd byte are illegal range,
IE 6 treat it as a two byte characters and convert to 30fb. we currently treat
it as single byte character and convert it to 0xfffd
c. for valid shift jis , if a character have no definitation . IE6 map it ot
30fb but we map to fffd

we need to fix all the three above so we have IE6 parity in error handling.

also, I wrote a cgi which generate legal shift according to the Nadin book also
invalide shift jis. I post in http://warp/u/ftang/utf8test/sjis.cgi
I will try to push it out to
http://people.netscape.com/ftang/testscript/sjis/sjis.cgi

Comment 16

[Frank Tang]

*** Bug 116880 has been marked as a duplicate of this bug. ***

Comment 17

[Frank Tang]

Attachment: patch v1

Comment 18

[Frank Tang]

add nhotta and shanjian to the list.

Comment 19

[Shanjian Li]

Comment on attachment 70437 [details] [diff] [review]
patch v1

r=shanjian,
(I suggest to remove the break in  original line 147.)

Comment 20

[nhottanscp]

+                   // IE convert fc-ff as single byte and convert to
+                   // U+f8f1 to U+f8f3
+                   if((0xfd == *src) || (0xfe == *src) || (0xff == *src))
+                   {
+                     *dest++ = (PRUnichar) 0xf8f1 + 
+                                   (*src - (unsigned char)(0xfd));

Does this mean, mapping like this? 
0xfd -> 0xf8f1
0xfe -> 0xf8f2
0xff -> 0xf8f3
But the comment says fc-ff (includes fc).

So is the IE6 behavior to map 0x30fb (the case c) specific to Shift_JIS or the
similiar behavior for EUC-JP?

Comment 21

[Frank Tang]

>Does this mean, mapping like this? 
>0xfd -> 0xf8f1
>0xfe -> 0xf8f2
>0xff -> 0xf8f3
>But the comment says fc-ff (includes fc).
good catch, it is fd-ff not fc. sorry. I will change the comment 

>So is the IE6 behavior to map 0x30fb (the case c) specific to Shift_JIS or the
>similiar behavior for EUC-JP?
Not sure, need develope more test. Let's fix it one by one. 
open bug 127275  for EUC-JP issue. 

Comment 22

[Frank Tang]

Attachment: patch v2

Comment 23

[Frank Tang]

nhotta or shanjian, please r= 

Comment 24

[nhottanscp]

Comment on attachment 70941 [details] [diff] [review]
patch v2 

r=nhotta

Comment 25

[kinmoz]

Comment on attachment 70941 [details] [diff] [review]
patch v2 

sr=[e-mail]

Comment 26

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Comment on attachment 70941 [details] [diff] [review]
patch v2 

a=roc+moz for 0.9.9

Comment 27

[Frank Tang]

fixed and check in.

Comment 28

[Teruko Kobayashi]

Verified as fixed in 0329 Win32 trunk and 0402 0.9.9ec Win32 build.

Comment 29

[Simon Montagu :smontagu]

Tests: http://hg.mozilla.org/mozilla-central/rev/fb086cc13695