181344 - Universal auto-detector detects western page as gb18030

Status: RESOLVED DUPLICATE of bug 177505

(Core :: Internationalization, defect)

Description

[Yuying Long]

Build: 11-20 trunk and 1.02 branch build.

I randomly saw a few nba.com sub-pages not display properly with Universal
auto-detector.

Steps to reproduce:
1. Launch browser with a new profile or cleared Cache.
2. Keep the default charaset as western ISO-8859-1.
3. View | Character Coding | Auto-detect, select Universal.
4. Load page: 
http://www.nba.com/mavericks/matchup/2001-02season_preview.html

Result:
The page doesn't display correctly with character " ' ", and the charset is
marked as gb18030.

Comment 1

[Yuying Long]

Attachment: saved today's page in URL

Comment 2

[Yuying Long]

Reassign to Shanjian for auto-detector issue.

Comment 3

[Mike Cowperthwaite]

An explanation of this bug is found in bug 177505 comment 5.

Comment 4

[Mike Cowperthwaite]

*** Bug 227933 has been marked as a duplicate of this bug. ***

Comment 5

[Mike Cowperthwaite]

*** Bug 205518 has been marked as a duplicate of this bug. ***

Comment 6

[Mike Cowperthwaite]

*** Bug 220555 has been marked as a duplicate of this bug. ***

Comment 7

[Mike Cowperthwaite]

There is a test page at that dupe which, when I just loaded it, was detected as 
ISO-8859-1.  But, there is also some possibly interesting troubleshooting info, 
including a bit of code analysis by S.Montagu.

Comment 8

[Grey Nicholson (they/them)]

This bug seems to be affecting http://www.hit-haven.com/z/zero7/destiny.htm

Using the universal encoding detector, the English text is interspersed with
Japanese characters.

The page is actually encoded as Western ISO-8859-1 (or something practically
equivalent) - manually switching to this encoding shows that the Japanese
characters appear in place of each right single quote (’).

Comment 9

[Grey Nicholson (they/them)]

(In reply to comment #8)

Similar shenanigans are to be found at
http://icwales.icnetwork.co.uk/0100news/0200wales/ (Japanese (Shift_JIS)) and
http://icwales.icnetwork.co.uk/0100news/ (Chinese Simplified (GB18030)) but not
at http://icwales.icnetwork.co.uk/ (Western (ISO-8859-1)).

Comment 10

[Mike Cowperthwaite]

The Shift_JIS problem is probably bug 168526.

Comment 11

[OstGote!]

*** Bug 274288 has been marked as a duplicate of this bug. ***

Comment 12

[Frank Tang]

shanjian is no longer working on mozilla for 2 years and these bugs are still
here. Mark them won't fix. If you want to reopen it, find a good owner first. 

Comment 13

[Travis Chase]

Mass Reassign Please excuse the spam

Comment 14

[Travis Chase]

Mass Re-opening Bugs Frank Tang Closed on Wensday March 02 for no reason, all
the spam is his fault feel free to tar and feather him

Comment 15

[Travis Chase]

Mass Re-assinging Frank Tangs old bugs that he closed won't fix and had to be
re-open. Spam is his fault not my own

Comment 16

[Jean-Marc Desperrier]

I'm currently trying to see what can be done to solve this bug.

I intend to work on both the SJIS and the gb18030 problem. I will open a thread
on netscape.public.mozilla.i18n to discuss my findings about the working of th
detector, and share idea about exaclty how it should be enhanced.

Shanjian says that he reports the 0x81~0xfe, 0x30~0x39, 0x81~oxfe, 0x30~0x39
immediately, but I didn't see that yet in the code, only the frequency detector,
whereas what he describes sounds more like a state machine.

Comment 17

[Gervase Markham [:gerv]]

Jean-Marc: did you get a chance to open that thread? I don't see it...

Gerv

Comment 18

[Richard Brodie]

*** Bug 291929 has been marked as a duplicate of this bug. ***

Comment 19

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

*** Bug 312071 has been marked as a duplicate of this bug. ***

Comment 20

[Phil Ringnalda (:philor)]

*** Bug 312691 has been marked as a duplicate of this bug. ***

Comment 21

[John G. Myers]

*** This bug has been marked as a duplicate of 177505 ***