132006 - Traditional Chinese is detected as Simplified Chinese

Status: VERIFIED FIXED

(Core :: Internationalization, defect)

Description

[Wong, Paul]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Win98; en-US; rv:0.9.9+) Gecko/20020318
BuildID:    20020318

Traditional Chinese is detected as Simplified Chinese

Reproducible: Always
Steps to Reproduce:
1.Go to http://pub16.ezboard.com/funionvillegeneralforum (for example)

Actual Results:  Mozilla selected Simplified Chiense Font

Expected Results:  Mozilla selected Traditional Chinese Font

Netscape 6.2.1 does select Traditional Chinese Font correctly.
and if I didn't remember wrongly, the early builds of 0.9.8 works correctly also.

Comment 1

[Arthur]

Confirming on Linux 2002031721. Autodetection displays simplified Chinese.

Comment 2

[Matthias Versen [:Matti]]

Arthur : Please select "Reassign bug to owner and QA contact of selected component"
if you cange the component...

Comment 3

[Yuying Long]

This particular page http://pub16.ezboard.com/funionvillegeneralforum
can be detect as big5 when select Auto-Detect Traditional Chinese, all other
auto-detect options will failed.

Comment 4

[Yuying Long]

On N6.2.1, this page is detected as big for auto-detect east asian, chinese but
auto-detect all still failed.

For auto-detect east asian and chinese, it's a regression.

Comment 5

[Roy Yokoyama]

-> shanjian

Comment 6

[Shanjian Li]

Attachment: patch

Comment 7

[Shanjian Li]

frank, this is a obvious mistake and it will cause regression to charset detector. 
Since this is a simple change, we might want to get this into branch. 

Comment 8

[nhottanscp]

nsbeta1-

We don't need to fix the Chinese detector for beta1.

Comment 9

[Shanjian Li]

This will cause regression to other detectors. Please reconsider. 

Comment 10

[Frank Tang]

Comment on attachment 78753 [details] [diff] [review]
patch

r=ftang

Comment 11

[Frank Tang]

let's seek sr= and land into trunk first. 

Comment 12

[Shanjian Li]

Mike, could you sr?

Comment 13

[Mike Shaver (:shaver emeritus)]

Comment on attachment 78753 [details] [diff] [review]
patch

sr=shaver.

Comment 14

[Shanjian Li]

fix checked into trunk. Keep this open for branch. 

Comment 15

[Yuying Long]

There is only one trunk (Mac) build availible today so far.

And I checked it on Mac10.1.3, it still shows only auto-detect TradChinese will
detect the page fine, all other modules will detect this page as iso-8859-2.

Comment 16

[Yuying Long]

Linux 04-17 trunk build has very similar result:
Detect as iso-8859-1 when auto-detect All, Chinese, East Asian.
Detect as big5 when auto-detect TradChinese.

Comment 17

[Shanjian Li]

The problem is that gb18030 covers all code points in big5. For big5 web pages, 
if there is more than one verifiers left, PSM detector will not make any
decision and default will be latin1. That raise a problem. I think what we
should do is to remove gb18030 if there is only 2 verifiers left. 

Comment 18

[Shanjian Li]

Attachment: additional patch

Comment 19

[Shanjian Li]

ftang/mike, could you r/sr my additional patch? 
It's necessary in order to make other charset verifiers (include japanese and
korean ones) work. Gb18030 covers too many code points. 

Comment 20

[Shanjian Li]

Attachment: test case

Comment 21

[Shanjian Li]

the original URL contains many single byte latin1 characters, such as 0xa0,
0xb7. It is an invalid test page. 

Comment 22

[Frank Tang]

Comment on attachment 79704 [details] [diff] [review]
additional patch

r=ftang
shanjian explained to me about this patch on phone. shanjian, please add
comment before the change line	to descript the reason as you said in the
phone.

Comment 23

[Shanjian Li]

Attachment: added comment to additional patch

Comment 24

[Frank Tang]

msanz and bobj all think this is an important issue to fix. let's nsbeta1+ it. 
this will impact the following detector
"East Asian" , "Chinese", "Simplified Chinese" detectors only.
RISK: Low risk. only local to those those users use these three detectors. 
[adt2], impact users- 
mainly chinese users- potential users- 55M (9.8% of total internet users)
also j and k users too (if they choose "East Asian" detector)

add additional 52.1 (9.2% ) and 25.2M (4.4%) users
sum of cjk are 131.8 M users ( 23.4% of total internet users)

Comment 25

[Shanjian Li]

*** Bug 138584 has been marked as a duplicate of this bug. ***

Comment 26

[Scott Collins]

Comment on attachment 79855 [details] [diff] [review]
added comment to additional patch

sr=scc

Comment 27

[Shanjian Li]

fix checked into trunk.

Comment 28

[Kai Lahmann (is there, where MNG is)]

*** Bug 131998 has been marked as a duplicate of this bug. ***

Comment 29

[Yuying Long]

Veified fixed on 04-25 trunk build.

Comment 30

[Asa Dotzler [:asa]]

Comment on attachment 79855 [details] [diff] [review]
added comment to additional patch

a=asa (on behalf of drivers) for checkin to the 1.0 branch

Comment 31

[wacheo]

seems that the problem still occurs in the 1.0 branch (build 2002050209).

Comment 32

[Frank Tang]

since this is fixed on trunk and verify on trunk, mark this bug as fixed. ylong,
plase mark this as verify on trunk.

Comment 33

[Yuying Long]

Add "verified on trunk" in status whiteboard.

Comment 34

[Shanjian Li]

*** Bug 143538 has been marked as a duplicate of this bug. ***

Comment 35

[Jaime Rodriguez, Jr.]

a= was given on 04/25, so we may need a refresh from drivers to get on the 1.0
branch.

Comment 36

[scottputterman]

adding adt1.0.0+ for checkin to the 1.0 branch.  Please ask for drivers approval
again since their last approval was more than 3 days ago and afterwards check
into the 1.0 branch.

Comment 37

[Yuying Long]

*** Bug 144540 has been marked as a duplicate of this bug. ***

Comment 38

[Roy Yokoyama]

*** Bug 144090 has been marked as a duplicate of this bug. ***

Comment 39

[Roy Yokoyama]

*** Bug 145218 has been marked as a duplicate of this bug. ***

Comment 40

[Roy Yokoyama]

*** Bug 145199 has been marked as a duplicate of this bug. ***

Comment 41

[Dawn Endico]

Comment on attachment 79855 [details] [diff] [review]
added comment to additional patch

a=chofmann,etc.

renewing approval from yesterday's driver meeting.

please check this in by midnight.

Comment 42

[Shanjian Li]

fix checked in to branch. 

Comment 43

[Yuying Long]

Verified fixed on 05-24 branch build on WinXP-SC and Mac OS 10.1.4.