115114 - autodetect universal detects french as Central European (ISO-8859-2)

Status: VERIFIED FIXED

(Core :: Internationalization, defect)

Description

[Jean-Marc Desperrier]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.6+)
Gecko/20011212
BuildID:    2001121203

autodetect universal is about unusable for French.

It wrongly identifies about systematically french page encoded in ISO-8859-1 as
ISO-8859-2.

I reported this as a browser bug, but there's the same problem in mail and news
everytime the content headers are missing.
In fact, it's most often encountered in mail/news as most web pages have the
ISO-8859-1 header.

"Auto-detect - all" in Netscape 6.2 has the same problem.

Reproducible: Always
Steps to Reproduce:
1. Select Universal in View/Character Coding/Auto-Detect
2. Load the sample URL http://www.aminautes.org/forums/configurer/oe/proe5.html

Actual Results:  
French users can tell immediatly some of the characters in the page are wrong. 
But if you don't know french, you can go in View/Character Coding, and see
Central European (ISO-8859-2) is shown selected. 
The content is truly in ISO-8859-1.

This happens about systematically with french text. 

I tried with some german, and it was also identified as Central European
(ISO-8859-2), but it just happens that all the german characters used in the
page (ö, ü, ä) have the same mapping in ISO-8859-1 and in ISO-8859-2.
Still the "normal" encoding of german is ISO-8859-1, and if the data is detected
as ISO-8859-2 in mail/news, the answer will be in ISO-8859-2, which might be
annoying.

[e-mail] is the owner of the universal autodetector.

Comment 1

[Jean-Marc Desperrier]

Attachment: a sample short french text that has the same problem

Comment 2

[Jean-Marc Desperrier]

Attachment: Another sample short french text that has the same problem

Comment 3

[Jean-Marc Desperrier]

Attachment: Another sample short french text that has the same problem

Comment 4

[Jean-Marc Desperrier]

Attachment: a sample short german text that is audetected as central european (the display happens to be correct)

The german umlaut characters are in fact common in ISO-8859-2 and ISO-8859-1,
and as a result the display is correct.
But the normal charset of german is ISO-8859-1, and it will be annoying for the
user that the data is identified is ISO-8859-2 instead.

Comment 5

[Jean-Marc Desperrier]

As the bug is completely reproductible in Netscape 6.2 whose implementation of
"autodetect-all" is independant from the implementation of
"autodetect-universal" in Mozilla, I wonder if the problem really is in the
autodetection module, or if there's not something wrong in the interpretation of
it's result that would map a result of ISO-8859-1 to ISO-8859-2.

Comment 6

[Shanjian Li]

I knew what cause the problem. Current algorithm for latinX detection need to 
be fine tuned. We do not try to detect latin1 charset and only default to it 
when all other detectors give us low confidence. Because latin1 and latin2 
charset share many code points, it is necessary to add latin1 detector and let 
it compete with latin2. I think most of the problem can be resolve this way. 

Comment 7

[Chris Kuklewicz]

This seems tangentially related a the bug I filed, #117758

http://bugzilla.mozilla.org/show_bug.cgi?id=117758

For some reason character E9 (hexidecimal) renders correctly with the
sample attached to this bug (115114) but on the front page of www.salon.com
it renders with a diamond w/ question mark.

Any ideas why?

Default Language English[en]and coding Western (iso-8859-1)

Comment 8

[Hugh Kennedy]

*** Bug 124734 has been marked as a duplicate of this bug. ***

Comment 9

[Yuying Long]

-> nsbeta1, since more similar bugs keep coming. 

Comment 10

[Frank Tang]

nsbeta1- since ns are not shipping with this.

Comment 11

[Jean-Marc Desperrier]

But there is a very similar bug in the "Auto-detect - all" feature of NS 
releases.

Comment 12

[Boris Zbarsky [:bzbarsky]]

*** Bug 129246 has been marked as a duplicate of this bug. ***

Comment 13

[Shanjian Li]

Attachment: patch

Comment 14

[Shanjian Li]

Charset latin1 and latin2 share many similarity, and it is pretty hard to
discriminate between each other. In current universal charset implementation,
there is not latin1 verifier, we only use latin1 as default. That's say if no
other charset verifiers declare confidence, we use latin1. That's works pretty
well until I added latin2 charset verifier. It seems like we have to  remove
this latin2 verifier before latin1 is available. Otherwise the possible
sideeffect is simpily too serious. 

Comment 15

[Shanjian Li]

roy, could you r= this patch?

Since the change is very simple, and impact is big, I would suggest to
reconsider this one for mozilla 1.0.  

Comment 16

[Shanjian Li]

*** Bug 131401 has been marked as a duplicate of this bug. ***

Comment 17

[Roy Yokoyama]

shanjian: 
I think we should ask Frank for review.
I am happy to review it for you if frank is too busy.

frank?

Comment 18

[Shanjian Li]

nsbeta1+ per ftang

roy, please go ahead and review it. I talked to frank and he had no objection. 

Comment 19

[Roy Yokoyama]

shanjian:
why do we need to remove
+//mProbers[11] = new nsSingleByteCharSetProber(&Win1250HungarianModel);
Did we have two latin2 charset verifiers?

Comment 20

[Shanjian Li]

win1250 should have the same problem as latin2. It may share some similarity
with win1252. Besides, we don't want have only one charset for Hungarian,
especially when it is not a popular one. 

Comment 21

[Jean-Marc Desperrier]

win1250 does have the same problem as latin2. 
In one of the duplicates, the ISO859-1 page was detected as win1250.
I didn't report specific cases, but I think I saw the same problem too.

A bit better change would be to have a preference in prefs.js that would enable
bulgarian users to reenable win1250 /latin2 autodetection if needed.
But it might be too late for that for 1.0 and not that useful compared to a
patch that would truly solve the problem.

Comment 22

[Frank Tang]

Comment on attachment 78373 [details] [diff] [review]
patch

r=ftang. remove sub module should not cause problem

Comment 23

[Shanjian Li]

brendan, could you sr?

Comment 24

[Frank Tang]

[adt2], this impact europen users. We want to use Universial detector for
Netscape beta also because the quality is better once we fix this problem 

Comment 25

[Scott Collins]

Comment on attachment 78373 [details] [diff] [review]
patch

sr=scc

Comment 26

[Shanjian Li]

fix checked into trunk.

Comment 27

[Yuying Long]

Verified on 04-19 on 3 platforms: WinME, Mac10.1.3 and linux RH7.2, the original
report page:
http://www.aminautes.org/forums/configurer/oe/proe5.html
is deletect as iso-8859-1 with auto-detect Universal.

Comment 28

[bruno lapeyre]

But under Mac OS X 10.1.4 the original report page:
http://www.aminautes.org/forums/configurer/oe/proe5.html
is detect as iso-8859-2 with auto-detect Universal.

Comment 29

[bruno lapeyre]

As I reported inside Bug 131401, this issue isn't occuring only with pages using
foreign characters. The problem sounds deeper. Even that pure american page is
detected as 8859-2 when Auto-detect is set to Universal:

http://maccentral.macworld.com/news/0204/19.sjaug.php

Many other american page can give the same result.
If this problem was a pure Mac OS X problem we probably should reopen Bug 131401...

Comment 30

[bruno lapeyre]

Please forget my Comments #28 and #29 : happily I was then using a previous
trunk. Sorry for that... Using the last trunk (2002041903) is giving excellent
results with the test page, with MacCentral pages and with each other pages
where the bug was detected. Great work.

Comment 31

[bruno lapeyre]

Attachment: Japanese encoding used when it shouldn't

Using the 2002O41903 build I discovered that when Auto Detect is set to
Universal, japanese encoding could be used when it should be western 8859-1 (I
already reported this issue before: see bug 131401 comment #26). The screenshot
was taken from the following page:
http://www.versiontracker.com/moreinfo.fcgi?id=13513&db=macosx

Comment 32

[Shanjian Li]

bruno lapeyre,
You might want to file a separate bug for that. Generally speaking, charset
autodetection will never be 100% accurate. That page contains some byte
combinations that can also be interpreted as a valid japanese character. So I am
not sure if that is solvable in near future.

Comment 33

[Frank Tang]

adk for adt1.0.0, QA have verify this on the trunk. 
The risk is low since we are removing the code inside that paticular detector
which are not turn on by default for English users. 

Comment 34

[Frank Tang]

some other bugs depend on this:
http://bugscape.mcom.com/show_bug.cgi?id=11459
http://bugscape.mcom.com/show_bug.cgi?id=11250
also, vantive 279691 is depend on this.

Comment 35

[scottputterman]

adding adt1.0.0+.  Please check this into the branch as soon as possible and add
the fixed1.0.0 keyword.

Comment 36

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 78373 [details] [diff] [review]
patch

a=[e-mail] for branch checkin

Comment 37

[Shanjian Li]

fix checked into trunk.

Comment 38

[Yuying Long]

Vefified works fine on 04-25 branch build.

Comment 39

[Michael Dunn]

Internal reference:
http://bugscape.netscape.com/show_bug.cgi?id=13722