23920 - Russian Auto detect detects character set wrong

Status: VERIFIED FIXED

(Core :: Internationalization, defect, P3)

Description

[Eugene Savitsky]

"Auto detect" detects the page like MacCyrillic. But the page is in WIN-1251.

Comment 1

[Frank Tang]

It seems we need to improve the russian detection algorithm. Mark this M18

Comment 2

[Frank Tang]

I think I fix this bug several weeks ago by adding a if( > 0x80) statement.

Comment 3

[Teruko Kobayashi]

I verified this in 2000022108 Win32, Mac, and Linux build.

Comment 4

[Eugene Savitsky]

win98 2000022508. This bug is back.

Comment 5

[Frank Tang]

Here is the fix
1. we should not report the charset if we collect no 8 bits data (all the prob
equal to 0)
2. we forget to compare index 0.

Index: src/nsCyrillicDetector.cpp
===================================================================
RCS file: /m/pub/mozilla/intl/chardet/src/nsCyrillicDetector.cpp,v
retrieving revision 1.6
diff -c -r1.6 nsCyrillicDetector.cpp
*** nsCyrillicDetector.cpp      2000/02/15 09:15:13     1.6
--- nsCyrillicDetector.cpp      2000/02/28 17:14:02
***************
*** 133,145 ****
     PRUint8 j;
     if(mDone)
        return;
!    for(j=1;j<mItems;j++) {
        if(mProb[j] > max)
        {
             max = mProb[j];
             maxIdx= j;
        }
     }
  #ifdef DEBUG
     for(j=0;j<mItems;j++)
        printf("Charset %s->\t%d\n", mCharsets[j], mProb[j]);
--- 133,149 ----
     PRUint8 j;
     if(mDone)
        return;
!    for(j=0;j<mItems;j++) {
        if(mProb[j] > max)
        {
             max = mProb[j];
             maxIdx= j;
        }
     }
+
+    if( 0 == max ) // if we didn't get any 8 bits data
+      return;
+
  #ifdef DEBUG
     for(j=0;j<mItems;j++)
        printf("Charset %s->\t%d\n", mCharsets[j], mProb[j]);

Comment 6

[Frank Tang]

I cannot check in this bug before beta since this is not our beta1 goal. 
(limited by Netscape's PDT rule). Move it to M16.

Comment 7

[Frank Tang]

*** Bug 30191 has been marked as a duplicate of this bug. ***

Comment 8

[Frank Tang]

*** Bug 30202 has been marked as a duplicate of this bug. ***

Comment 9

[Frank Tang]

Change the summary to "Russian Auto detect detects character set wrong". Move to 
m15

Comment 10

[Frank Tang]

fix check in.

Comment 11

[Frank Tang]

*** Bug 31954 has been marked as a duplicate of this bug. ***

Comment 12

[Teruko Kobayashi]

Verified as fixed in 2000041109 Win32 build.