343465 - In Russian spell-checker not supported cyrillic letter "yo"

Status: RESOLVED FIXED

(Mozilla Localizations :: ru / Russian, enhancement)

Description

[Arkady Belousov]

User-Agent:       Opera/9.00 (Windows 98; U; ru)
Build Identifier: http://mozilla-chi.osuosl.org/pub/mozilla.org/thunderbird/releases/1.5.0.4/win32/ru/Thunderbird%20Setup%201.5.0.4.exe

I take look into http://downloads.mozdev.org/dictionaries/spell-ru.xpi and found, that it identical to
http://ftp.services.openoffice.org/pub/OpenOffice.org/contrib/dictionaries/ru_RU.zip
(dictionary without "yo" support), but not to
http://ftp.services.openoffice.org/pub/OpenOffice.org/contrib/dictionaries/ru_RU_yo.zip
(dictionary with "yo" support).

Why this is important? In Russia exists unofficial tradition sometime replace "yo" by "ye" (especially, for example, because keyboard of old imported typewritting machines was not support this letter). But not all peoples do this (they prefer to write in clean, unambiguous language), but current distributed spellchecker doesn't support correct language.

Reproducible: Always

Comment 1

[Mike Cowperthwaite]

For those unfamiliar with Russian/Cyrillic:
  "ye" = 'е' U+0435  (looks like ASCII 'e')
  "yo" = 'ё' U+0451  (looks like e-with-umlaut)

By '"yo" support' do you mean the existing dictionary allows "ye" to be read as "yo", or that it *doesn't* allow it, instead forces "yo" always to be used?  

Either way: It appears you are asking for the capability to pick which dictionary to use, as Open Office does -- correct?

Comment 2

[Arkady Belousov]

> For those unfamiliar with Russian/Cyrillic:
>   "ye" = 'е' U+0435  (looks like ASCII 'e')
>   "yo" = 'ё' U+0451  (looks like e-with-umlaut)

     "yo" is e-with-horizontal-colon-above-it. Also, don't know about other code pages and unicode, but in CP1251 "ye" have codes 0xC5 and 0xE5 (upper lower), whereas "yo" - 0xA8 and 0xB8. In CP866 these codes are 0x85/0xA5 and 0xF0/0xF1.

> By '"yo" support' do you mean the existing dictionary allows "ye" to be read as
> "yo", or that it *doesn't* allow it, instead forces "yo" always to be used?  

     By "yo support" I mean accepting words, where should be used "yo".

     Accepting "ye" instead "yo" is another question. And _I myself_ prefer to use dictionary, where replacement is not supported, because this will force using only valid writing. Especially because, as I say, some words with "ye" and "yo" are _different_ words - for example, "vse" and "vse:" mean "everybody" and "everything".

> Either way: It appears you are asking for the capability to pick which
> dictionary to use, as Open Office does -- correct?

     Yes, this also may be solution.

Comment 3

[Arkady Belousov]

> > By '"yo" support' do you mean the existing dictionary allows "ye" to be read as
> > "yo", or that it *doesn't* allow it, instead forces "yo" always to be used?  
>      By "yo support" I mean accepting words, where should be used "yo".

     Ie., in dictionary, which currently distributed, all "yo" occurences replaced by "ye". This is sometime acceptable, but this is wrong.

Comment 4

[Justin Forest]

This is not an enhancement, this is a bug.  Russian Language has this letter; a dictionary which doesn't have this letter but claims to be Russian, isn't.  The absence of this letter wreaks havoc on official documents (such as passports), where exact spelling matters, which is why the letter YO was recently restored and is a mandatory for all official documents in Russia.

Moreover, when you check a perfectly correct document with this incomplete dictionary, it brings up a huge number of false errors.  What does an average user do?  Disable the spell checker.

Comment 6

[Alexander L. Slovesnik]

(In reply to comment #0)
> http://ftp.services.openoffice.org/pub/OpenOffice.org/contrib/dictionaries/ru_RU_yo.zip
The reason that this dictionary wasn't used in Russian builds of Firefox/Thunderbird is that it's license is not clear.
Mozilla CVS can contain only MPL/GPL/LGPL or BSD-licensed code. License of ru_RU_yo looks close to BSD license, but I'm not a lawyer.

Comment 7

[Alexander L. Slovesnik]

Moving to right component.

Comment 8

[Arkady Belousov]

After installing TB 2.0 (over 1.5), I found that anything is worser: there were no dictionaries available in Tools/Composition/Spellcheck, even though on disk there were at least three instances in different places (%TBirdDir%\dictionaries, %TBirdDir%\components\myspell and somewhere more). Now I forced to install dictionaries as plugins (see screenshot) to make them available (now russian dictionary duplicated in %AppDir%\%TBird%).

PS: BTW, this plugin contains dictionary with "yo", so given report may, probably, be closed (if not count, that dictionary was not available after installation). Also, there remains ALL problems with spellchecking from 1.x branch - for example, no possibility to use more than one dictionary simultaneously.

Comment 9

[Arkady Belousov]

Attachment: Dictionaries installed manually to make them available

Comment 10

[Alexander L. Slovesnik]

(In reply to comment #8)
> After installing TB 2.0 (over 1.5), I found that anything is worser: there were
> no dictionaries available in Tools/Composition/Spellcheck, even though on disk
> there were at least three instances in different places
> (%TBirdDir%\dictionaries, %TBirdDir%\components\myspell and somewhere more).
I think you should erase Thunderbird 1.5 and install 2.0 in clean directory. Probably some remains of Thunderbird 1.5 left, that keeps Thunderbird 2.0 from working.

> PS: BTW, this plugin contains dictionary with "yo", so given report may,
> probably, be closed (if not count, that dictionary was not available after
> installation).
I prefer to keep this bug open to track issues with "yo" letter. We have several options to resolve this bug:
1) Remove current "without yo" dictionary from build and tell users to install "with yo" dictionary as extension.
2) Improve current "without yo" dictionary and include "yo" support in it.
3) Include "with yo" dictionary (if license of this dictionary is compatible with Mozilla). We need some legal experts here. I already sent letter to Gervase Markham and asked him to take a look at "with yo" dictionary license.

I personally prefer 3).

Comment 11

[Arkady Belousov]

> I think you should erase Thunderbird 1.5 and install 2.0 in clean directory.
> Probably some remains of Thunderbird 1.5 left, that keeps Thunderbird 2.0 from
> working.

- too late - I already install TB2 over TB1.

- in theory, there shouldn't be problems with installing one versions over others. If someone enounter problem with installing TB2 over TB1, then this is definitely bug in TB, which should be fixed (especially, I not see much of the differences between TB2 and TB1 - even almost all bugs of TB1 are remained in TB2). And if there are unavoidable incompatibilities, installer should prevent installation over previous versions and should offer import from it.

Fortunately, TB2 doesn't "stop working" in my installation, there is just problem that after TB2 installation there was no dictionaries available to select.

> > probably, be closed (if not count, that dictionary was not available after
> I prefer to keep this bug open to track issues with "yo" letter. We have
> several options to resolve this bug:
> 1) Remove current "without yo" dictionary from build and tell users to install
> "with yo" dictionary as extension.

In this case, installer should explicitly notify about this and give link for download.

> 2) Improve current "without yo" dictionary and include "yo" support in it.
> 3) Include "with yo" dictionary (if license of this dictionary is compatible
> with Mozilla). We need some legal experts here. I already sent letter to
> Gervase Markham and asked him to take a look at "with yo" dictionary license.
> I personally prefer 3).

I prefer 3) too - "relaxed ye-dictionary" should be available, but only as option (external plugin). Especially, Justin Forest (see comment 4) is right, status of "yo" is now step-by-step restored.

PS: Now, with spell-checking in TB there remains two problems for me:

- no support for multi-language texts (in my case, mix of Russian and English);
- no support for descripting derivatives when adding unknown words;
- no support for grammar/syntax checking (should be used more advanced engines and dictionaries).

But this is issue for other reports.

Comment 12

[Arkady Belousov]

> I think you should erase Thunderbird 1.5 and install 2.0 in clean directory.
> Probably some remains of Thunderbird 1.5 left, that keeps Thunderbird 2.0 from
> working.

- too late - I already install TB2 over TB1.

- in theory, there shouldn't be problems with installing one versions over others. If someone enounter problem with installing TB2 over TB1, then this is definitely bug in TB, which should be fixed (especially, I not see much of the differences between TB2 and TB1 - even almost all bugs of TB1 are remained in TB2). And if there are unavoidable incompatibilities, installer should prevent installation over previous versions and should offer import from it.

Fortunately, TB2 doesn't "stop working" in my installation, there is just problem that after TB2 installation there was no dictionaries available to select.

> > probably, be closed (if not count, that dictionary was not available after
> I prefer to keep this bug open to track issues with "yo" letter. We have
> several options to resolve this bug:
> 1) Remove current "without yo" dictionary from build and tell users to install
> "with yo" dictionary as extension.

In this case, installer should explicitly notify about this and give link for download.

> 2) Improve current "without yo" dictionary and include "yo" support in it.
> 3) Include "with yo" dictionary (if license of this dictionary is compatible
> with Mozilla). We need some legal experts here. I already sent letter to
> Gervase Markham and asked him to take a look at "with yo" dictionary license.
> I personally prefer 3).

I prefer 3) too - "relaxed ye-dictionary" should be available, but only as option (external plugin). Especially, Justin Forest (see comment 4) is right, status of "yo" is now step-by-step restored.

PS: Now, with spell-checking in TB there remains two problems for me:

- no support for multi-language texts (in my case, mix of Russian and English);
- no support for descripting derivatives when adding unknown words;
- no support for grammar/syntax checking (should be used more advanced engines and dictionaries).

But this is issue for other reports.

Comment 13

[Alexander L. Slovesnik]

(In reply to comment #11)
> Fortunately, TB2 doesn't "stop working" in my installation, there is just
> problem that after TB2 installation there was no dictionaries available to
> select.
Have you installed Thunderbird 2.0 or 2.0.0.4? May be you runned into Bug 367533?
 
> PS: Now, with spell-checking in TB there remains two problems for me:
> - no support for multi-language texts (in my case, mix of Russian and English);
Russian community has made two russian-english dictionaries - see http://ftp.mozilla-russia.org/dictionaries/

Comment 14

[Arkady Belousov]

> > Fortunately, TB2 doesn't "stop working" in my installation, there is just
> Have you installed Thunderbird 2.0 or 2.0.0.4? 

     AFAIR, this was TB 2.0.0.1. Now it auto-updated to 2.0.0.6.

> May be you runned into Bug 367533?

     I not completely remember all troubles, which I was encounter in Options dialog related to missing dictionaries (which TB not found in own directories), but I not remember screen glitches, which discussed in Bug 367533.

> > - no support for multi-language texts (in my case, mix of Russian and English);
> Russian community has made two russian-english dictionaries - see
> http://ftp.mozilla-russia.org/dictionaries/

     Do you mean http://ftp.mozilla-russia.org/dictionaries/ru-en_spell_dictionary.xpi ? Nice. Not perfect (I myself prefer writing only with "yo"), but as temporary solution this should be enough.

Comment 15

[Alexander L. Slovesnik]

(In reply to comment #14)
> > Russian community has made two russian-english dictionaries - see
> > http://ftp.mozilla-russia.org/dictionaries/
> 
>      Do you mean
> http://ftp.mozilla-russia.org/dictionaries/ru-en_spell_dictionary.xpi ? Nice.
> Not perfect (I myself prefer writing only with "yo"), but as temporary solution
> this should be enough.
Yes. And http://ftp.mozilla-russia.org/dictionaries/ru-us-dictionary.xpi too. 

Comment 16

[Alexander L. Slovesnik]

(In reply to comment #10)
> 3) Include "with yo" dictionary (if license of this dictionary is compatible
> with Mozilla). We need some legal experts here. I already sent letter to
> Gervase Markham and asked him to take a look at "with yo" dictionary license.
> 
> I personally prefer 3).
I filed Bug 389071 for including "with yo" dictionary in Russian builds. 

Comment 17

[Arkady Belousov]

(In reply to comment #15)
> > http://ftp.mozilla-russia.org/dictionaries/ru-en_spell_dictionary.xpi ?
> Yes. And http://ftp.mozilla-russia.org/dictionaries/ru-us-dictionary.xpi too. 

Second dictionary 30% shorter. I don't know, what missing here - but even first dictionary doesn't contains some words.

Comment 18

[Alexander L. Slovesnik]

Fixed in Bug 389071