128580 - Shift-JIS, EUC-JP pages without charset info don't load under ISO-2022-JP encoding

Status: VERIFIED FIXED

(Core :: Internationalization, defect, P3)

Description

[Katsuhiko Momoi]

** Observed with 2002-03-01 Win 32 trunk build **

I will attach a test page next -- Netscape Japan Home Page with the meta
charset info line deleted.  

Basically the problem is this:

When the browser default encoding is set to ISO-2022-JP, unless
the page is marked with a meta-charset info or the server sends HTTP
charset info, neither Shift_JIS or EUC-JP Japanese page loads.

See the test page first and the above URL.

Comment 1

[Katsuhiko Momoi]

Attachment: Netscape Japan Home Page without the meta charset tag

This is basically the Netscape Japan Home Page without the images and 
meta-charset info for x-sjis.
Load this page without Auto-Detection and with the browser encoding
default set to Japanese (ISO-2022-JP).

Comment 2

[Katsuhiko Momoi]

Let me explain the above URL. This is a more realistic example.

The user sets the following defaults:

browser encoding: ISO-2022-JP
Auto-Detect: Japanese ON

Now go visit the above URL (http://quuqai.tripod.co.jp/). Nothing loads.

The reason seems to be as follows. There is the top page and then 2 sub-frames.
None of the pages carry document charset info. The top page has some 
Shift_JIS characters but not enough for auto-detection. So the default
kicks in --> ISO-2022-JP.
The frame documents use Shift_JIS and they don't load under ISO-2022-JP.

Thus you get a blank display.

There are other examples of this same problem. Try to follow any links
from this top page:

http://www.kt.rim.or.jp/~nor/


You get a blank display. 

I understand that we cannot display Shift_JIS or EUC-JP pages under 
ISO-2022-JP encoding well. But blank display like this would be
very bad. 
I am filing this bug to request that we shoud at least load 
Shift_JIS or EUC-JP documents under ISO-2022-JP, not a blank display.

Note that other browsers don't allow ISO-2022-JP to be in the menu. 
Thus it is not possible to set that as the default. Mozilla allows it
and then we should live with this type of problem somewhat better.

Comment 3

[Katsuhiko Momoi]

I can foresee criticisms on this bug from engineers. So
let me explain a bit more on my motivation.

1. Average users probably don't know that it is practically useless to
   choose ISO-2022-JP as the default if Japanese Auto-Detection in ON.
2. But ISO-2022-JP is available as a default choice on the list.
3. Suppose that your Intranet uses ISO-2022-JP as the web page encoding.
   There are some companies with this practice. You might think, first
   set the JPN auto-detection. Then also ISO-2022-JP as the default 
   because that is the encoding used on your company pages.
4. Then you go to a real page like the above URL and other
   pages I cited. Auto-Detect fails because the top page that loads frames
   has very few characters. You get a blank display for frames which
   are encoded in Shift_JIS (or EUC-JP).

I can make 3 recommendations to avoid a case like this:

A. Maybe we should eliminate ISO-2022-JP from the default encoding list
   for browsing.
B. We should display something but not blank in this type of case.
C. We should throw up a warning dialog. That the user should change 
   the encoding to Shift_JIS or EUC-JP.

Please note that this kind of problem will not happen if the encoding
choice is not ISO-2022-JP. ISO-2022-JP may be special in this regard.
So option C or A may not be bad choice if we cannot do option B.

Comment 4

[Katsuhiko Momoi]

Well, this looks like a regression from 0.9.4. NS 6.2.1 
loads the frames in the above URLs and other examples I cited.
Also first test case I uploaded also loads. 
But the current trunk build does not load anything in these
examples, so this seems to be a regression. Ithink we should fix this
problem -- option B above in comment #3. 

Comment 5

[Katsuhiko Momoi]

Mozilla Japan contributor noted in the following Japanese bug:

http://bugzilla.mozilla.gr.jp/show_bug.cgi?id=1951

that this problem started happening with 2002-02-27 trunk build.

So this is a very recent regression.

Comment 6

[Roy Yokoyama]

This is really bad.
Just visit http://bugzilla.mozilla.gr.jp/show_bug.cgi?id=1951
and change the char encoding to 'ISO-2022-JP'.

The page is BLANK!!!

Comment 7

[Frank Tang]

reassign to ftang
nsbeta1+

Comment 8

[Frank Tang]

assign

Comment 9

[Frank Tang]

P3 

Comment 10

[Frank Tang]

Attachment: patch v1., return the right error code.

what happen is the return code we used are not a error code but a success code
(changed by nhotta in r1.9) the patch return the same error code we return in
utf8  converter.

Comment 11

[nhottanscp]

Comment on attachment 75048 [details] [diff] [review]
patch v1., return the right error code. 

r=nhotta

Comment 12

[Alec Flett]

Comment on attachment 75048 [details] [diff] [review]
patch v1., return the right error code. 

sr=alecf

Comment 13

[Asa Dotzler [:asa]]

Comment on attachment 75048 [details] [diff] [review]
patch v1., return the right error code. 

a=asa (on behalf of drivers) for checkin to the 1.0 trunk

Comment 14

[Frank Tang]

fixed and checkin 

Comment 15

[Yuying Long]

Fixed verified on 03-25 trunk build / WinXP-SimpChinese.