471799 - Hunspell doesn't recognize misspelled words if they are in different encoding

Status: RESOLVED WORKSFORME

(Core :: Spelling checker, defect)

Description

[Rail Aliiev [:rail]]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2a1pre) Gecko/20090101 Minefield/3.2a1pre Ubiquity/0.1.4
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2a1pre) Gecko/20090101 Minefield/3.2a1pre Ubiquity/0.1.4

Hunspell cannot properly handle words which use encoding not "compatible" with the current dictionary encoding. At least this is true for inline spell checking.

Reproducible: Always

Steps to Reproduce:
1. Set the current dictionary to en-US (which uses ISO8859-1)
2. Write a _wrong_ word which uses non-western symbols. For example, I use Russian word "тестх" (right one is "тест")

Actual Results:  
Wrong word is not underlined.

Expected Results:  
Wrong word should be underlined.

The problem can be worked around if we change the encoding of en-US dictionary (s/SET ISO8859-1/SET UTF-8/ and recode if needed).
Not investigated, but I think, there are some lost after character conversion within hunspell module.
BTW, hunspell used in OpenOffice.org returns the expected result.

Comment 1

[Alexander L. Slovesnik]

Confirmed with Mozilla/5.0 (X11; U; Linux i686; ru; rv:1.9.2a1pre) Gecko/20081226 Minefield/3.2a1pre ID:20081226103856

Comment 2

[Alexey Gladkov]

s/SET ISO8859-1/SET UTF-8/ definitely not enough. I get the following errors with this change: 

This UTF-8 encoding can't convert to UTF-16:
smörgåsbord
UTF-8 encoding error. Missing continuation byte in 5. character position:
soigné

So definitely need to recode.

Comment 3

[Kagami Rosylight [:saschanaz] (they/them)]

It seems this is in multiple language design area in some way, and I think it works now if I understand this correctly.

Comment 4

[Dan Minor [:dminor]]

The behaviour was changed in Bug 1773802 when multiple dictionaries are enabled. If only a single dictionary is enabled, we still don't spellcheck words in using other encodings. This was a deliberate choice to not change the behaviour for users with a single dictionary.