228545 - Bug "FilterWith..." in universal charset detector

Status: RESOLVED FIXED

(Core :: Internationalization, defect)

Description

[Vladimir]

User-Agent:       Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) Opera 7.23  [en]
Build Identifier: 

Two bugz nsSBCSGroupProber.cpp 
1. In FilterWithoutEnglishLetters need to be added (Line 133) after loop

if (meetMSB && curPtr > prevPtr) 
	while (prevPtr < curPtr) *newptr++ = *prevPtr++;  

To keep last word in text chunk

2. Same bugfix for FilterWithEnglishLetters (Line 173) after loop
if (curPtr > prevPtr && !isInTag) //current segment contains more than just a 
symbol 
                                    // and it is not inside a tag, keep it
  while (prevPtr < curPtr) *newptr++ = *prevPtr++;

Reproducible: Always

Steps to Reproduce:
1. Try single non english word like ???????? (one word) - You'll lose it in 
FilterWithoutEnglishLetters 
2. Try single english word - it will be losed in FilterWithEnglishLetters

Comment 1

[Vladimir]

Attachment: patch for 228545 bug

Comment 2

[Vladimir]

Attachment: bug reproduction file

FilterWithoutEnglishLetters loose last word
(потеряно) in first line
FilterWithEnglishLetters loose last word (lostword) in second line

Comment 3

[Simon Montagu :smontagu]

Comment on attachment 151520 [details] [diff] [review]
patch for 228545 bug

r=smontagu

Comment 4

[Mike Kaply [:mkaply]]

Can you attach a -u patch for this one so that someone can check it in for you
and I can look at it better?

Comment 5

[Vladimir]

(In reply to comment #4)
> Can you attach a -u patch for this one so that someone can check it in for you
> and I can look at it better?
With pleasure but tomorrow. My fast inet and fixed version in office :-(

Comment 6

[timeless]

mozilla/extensions/universalchardet/src/nsSBCSGroupProber.cpp 	1.11

mkaply:
http://tinderbox.mozilla.org/bonsai/multidiff.cgi?allchanges=timeless%
mozdev.org|mozilla/extensions/universalchardet/src|nsSBCSGroupProber.cpp|1.11,&c
vsroot=/cvsroot

Comment 7

[Mike Kaply [:mkaply]]

Comment on attachment 151520 [details] [diff] [review]
patch for 228545 bug

a=mkaply for 1.7.1

Comment 8

[timeless]

mozilla/extensions/universalchardet/src/nsSBCSGroupProber.cpp 	1.9.14.1 	

Comment 9

[Mike Cowperthwaite]

Comment 6 and comment 8, and the 'fixed1.7.2' keyword, imply this patch has been 
checked in; is there some reason the bug has not been marked FIXED?

Timeless?

Comment 10

[Vladimir]

Thanx to everybody for help in application of this patch