1402822 - Spellchecker needs to allow multiple language dictionaries at the same time

Status: RESOLVED FIXED

(Core :: Spelling checker, enhancement, P3)

Description

[Ben Bucksch (:BenB)]

Problem:
I routinely write emails in 3 different languages. They are all mixed: one email in English, another in German, and some in French. In some cases, I even write 2 different languages to the same person, depending on who else is or was involved in the thread, or depending on the subject.

Importance:
The problem arises for all users who speak several languages. E.g. they speak their local language with their friends, and English with international correspondents. In other words, this bug is important for all international communication (unless your native language happens to be English).

Non-solutions:
I cannot specify the language for each email. That costs too much time, and I'll rather ignore the spell-checker altogether.

Solution:
The only solution is to once - as a configuration - select a number of languages that I know how to write, and then to allow all words in these languages.

This is what Android keyboard does as well.

Out of scope:
If we can automatically detect the language that the user writes, that would be nice, but that would be another bug, an enhancement on top of this bug here.

Etiquette:
*****************************************************
*** Comments from end-users are not allowed here. ***
*****************************************************
These should be posted in bug 69687 instead, as they would be distracting here. Thanks for your understanding. This is an engineering implementation bug. But if you are waiting for this feature and want it to be implemented fast, please vote for this bug here.

Comment 1

[Dawid Młynarczyk]

Just to add my two cents and share some user experience... Constant-manual-language-switching is getting so annoying, that I started to consider either turning spell checks off completely in Firefox, or switching to Chrome. I was trying to deal with it, until I was using mainly Polish and English (+ some Spanish or German from time to time). But 3 years ago I met my future wife from Slovakia, and this year we got married. Switching between 3 languages is really tedious work... Plus I'm still learning Slovak, and she is learning Polish, so in communication between us we mix all 3 languages quite frequently if not constantly... ;]

Comment 2

[Gingerbread Man]

How does this differ from bug 481884 that you haven't touched since 2014?

(In reply to Ben Bucksch (:BenB) from comment #0)
> If we can automatically detect the language that the user writes, that would
> be nice, but that would be another bug

Bug 1203024

> *** Comments from end-users are not allowed here. ***

You can ask someone with the right privileges to restrict comments.

Comment 6

[Alexander Popov]

> Duplicate of this bug: 481884

Is the older bug duplicate newer? Okay...

Comment 8

[Ben Bucksch (:BenB)]

Chrome has option "All your languages" (combined): https://bugzilla.mozilla.org/attachment.cgi?id=8845416
Chrome puts different language suggestions together: https://bugzilla.mozilla.org/attachment.cgi?id=9075388

So, Chrome fixed exactly this bug here, in the same way as suggested here.

This is arguably the correct solution. I often mix words from other languages in a sentence.
This is also the easiest solution, because you don't need a complicated and error-prone language detection. You simply make the dictionary the super set of all the languages (typically 2 or 3) that the user selected.

Comment 9

[Martijn]

My current work-around is adding ~100K of the most common words from my native dictionary to the 'persdict.dat' file (within your Firefox profile folder). Works OK without any noticeable performance penalty, but I (and probably many others) would definitely love to see multi-language spell checking in Firefox. This is a feature Chrome has since 47.

Comment 10

[Carlo]

Hello, I'd like to work on this bug. I've already developed a possible solution and would like some feedback.

Can someone assign this bug to me and guide me through the patch process?

Comment 11

[Ben Bucksch (:BenB)]

Carlo: Happy to do that! I've assigned the bug to you.
In case you stop working on it, without the fix actually landing in the code tree, please unassign the bug again.
If you have a patch, be sure to ask for review while you attach the patch. The dropdrown should normally suggest appropriate reviewers.
I've also sent you a welcome message with some additional help.

Comment 12

[Emilio Cobos Álvarez (:emilio)]

You can ask any question related to patch submission / etc in https://chat.mozilla.org/#/room/#introduction:mozilla.org

There are other links in the topic of that channel which you may find helpful.

Comment 13

[cOgnaut]

Hi, I'd just like to add my vote to please fix this issue. Technically it's a feature request, but it's a feature that is sorely missed by many, many people around the world.

Actually, Chrome's solution to this is quite elegant: the spellcheck language is set not by a radio button, but by check boxes, and apparently, the browser simply "combines" all selected languages when spellchecking (rather than spellchecking only in the 1 selected language).

Comment 14

[Carlo]

(In reply to cOgnaut from comment #13)

Actually, Chrome's solution to this is quite elegant: the spellcheck language is set not by a radio button, but by check boxes, and apparently, the browser simply "combines" all selected languages when spellchecking (rather than spellchecking only in the 1 selected language).

This is what I implemented, but I have been really busy recently and don't have the time to actually finish working on it, for now.
Especially because on my macbook it takes ages to compile.

I hope to get some free time to finally finish the patch and post it here for review.

Comment 15

[Ben Bucksch (:BenB)]

This is what I implemented

Can you share the code, please?

Comment 16

[Carlo]

Attachment: 0001-Draft-patch-for-bug-1402822.patch

Here it is. This is after a git pull --rebase.
Unfortunately, mach tells me there's something in my tree that doesn't let it compile, so I don't even know if it compiles now. I would have to checkout everything again from scratch and run a full compile, but it would take hours.

The code worked when I wrote it, the patch was missing a couple of failing tests to be fixed and general polishing / splitting in multiple smaller commits.

The result was that the dictionary selection menu was not a radio group anymore but a multiple select and misspelling suggestions would come out in round robin for each message, because hunspell doesn't return a score to sort by.

Comment 17

[daniel]

(In reply to Carlo from comment #16)
thanks for your contribution Carlo.

Comment 19

[Ben Bucksch (:BenB)]

🎉
@Dan Minor: Thank you for working on this!

Comment 20

[kocjs]

I am looking for this too.

Comment 21

[Dan Minor [:dminor]]

Attachment: Bug 1402822 - Support multiple dictionaries in mozISpellCheckingEngine.idl; r=smaug!

Comment 22

[Dan Minor [:dminor]]

Attachment: Bug 1402822 - Support multiple hunspell instances in mozHunspell; r=smaug!

Depends on D140238

Comment 23

[Dan Minor [:dminor]]

Attachment: Bug 1402822 - Support multiple dictionaries in mozSpellChecker; r=smaug!

Depends on D140239

Comment 24

[Dan Minor [:dminor]]

Attachment: Bug 1402822 - Support multiple dictionaries in RemoteSpellcheckEngine; r=smaug!

Depends on D140240

Comment 25

[Dan Minor [:dminor]]

Attachment: Bug 1402822 - Support multiple dictionaries in InlineSpellChecker; r=smaug!,gijs!

Depends on D140241

Comment 26

[Dan Minor [:dminor]]

Attachment: Bug 1402822 - Support multiple dictionaries in EditorSpellCheck; r=smaug!

Depends on D140242

Comment 27

[Dan Minor [:dminor]]

Attachment: Bug 1402822 - Fix dictionary extension loading code in XPIProvider.jsm; r=smaug!

Depends on D140243

Comment 28

[Dan Minor [:dminor]]

Attachment: Bug 1402822 - Update unit tests for multiple dictionaries; r=smaug!

For the most part these are simple updates to account for multiple
dictionaries and the fact that SetDictionaries is async whereas
SetDictionary was sync.

Fixing test_async_UpdateCurrentDictionary was more involved because
there were flaws in the existing test and it is more difficult to
harmonize the UpdateCurrentDictionary callback with the changes to make
SetDictionaries async.

Depends on D140244

Comment 29

[Dan Minor [:dminor]]

Attachment: Bug 1402822 - Add mochitest for multiple dictionaries; r=smaug!

Depends on D140245

Comment 31

[Dan Minor [:dminor]]

There's failures on test-verification with these changes, but there's failures on test-verification on these tests without the changes as well, e.g. https://treeherder.mozilla.org/jobs?repo=try&revision=99fedcc1a058b3377c7def7938d3f27173edaee6 where I added some comments to trigger test-verification on the existing tests.

Fixing https://bugzilla.mozilla.org/show_bug.cgi?id=1760824 should help make the tests less flaky. I've tried to improve them a bit as part of these changes as well.

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/264664a44f49
Support multiple dictionaries in mozISpellCheckingEngine.idl; r=smaug
https://hg.mozilla.org/integration/autoland/rev/1af11d34c213
Support multiple hunspell instances in mozHunspell; r=smaug
https://hg.mozilla.org/integration/autoland/rev/043d88291491
Support multiple dictionaries in mozSpellChecker; r=smaug
https://hg.mozilla.org/integration/autoland/rev/4a3b7ee6a51e
Support multiple dictionaries in RemoteSpellcheckEngine; r=smaug
https://hg.mozilla.org/integration/autoland/rev/74f061f43c77
Support multiple dictionaries in EditorSpellCheck; r=smaug
https://hg.mozilla.org/integration/autoland/rev/338485f1d803
Support multiple dictionaries in InlineSpellChecker; r=smaug,Gijs
https://hg.mozilla.org/integration/autoland/rev/64b43b5b1bba
Fix dictionary extension loading code in XPIProvider.jsm; r=smaug,extension-reviewers,mixedpuppy
https://hg.mozilla.org/integration/autoland/rev/e86c493e43a4
Update unit tests for multiple dictionaries; r=smaug
https://hg.mozilla.org/integration/autoland/rev/465a18abbd0e
Add mochitest for multiple dictionaries; r=smaug

Comment 33

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/264664a44f49
https://hg.mozilla.org/mozilla-central/rev/1af11d34c213
https://hg.mozilla.org/mozilla-central/rev/043d88291491
https://hg.mozilla.org/mozilla-central/rev/4a3b7ee6a51e
https://hg.mozilla.org/mozilla-central/rev/74f061f43c77
https://hg.mozilla.org/mozilla-central/rev/338485f1d803
https://hg.mozilla.org/mozilla-central/rev/64b43b5b1bba
https://hg.mozilla.org/mozilla-central/rev/e86c493e43a4
https://hg.mozilla.org/mozilla-central/rev/465a18abbd0e

Comment 34

[Sylvestre Ledru [:Sylvestre]]

Release Note Request (optional, but appreciated)
[Why is this notable]: Great new feature for polyglots
[Affects Firefox for Android]: I don't think so?
[Suggested wording]: The spell checker allows multiple language dictionaries at the same time
[Links (documentation, blog post, etc)]:

Comment 35

[Dianna Smith [:diannaS]]

Release note added, please see https://www.mozilla.org/firefox/100.0a1/releasenotes/

Comment 36

[Alexander Gavrilov [:dartraiden]]

Has the about:config → spellchecker.dictionary setting been adapted to support multiple dictionaries? In my tests setting it to something comma-separated like "ru,en-US" does nothing.

Comment 38

[Dan Minor [:dminor]]

(In reply to Alexander Gavrilov from comment #36)

Has the about:config → spellchecker.dictionary setting been adapted to support multiple dictionaries? In my tests setting it to something comma-separated like "ru,en-US" does nothing.

We first try to match the element or document language and then use this preference. So if we think the input is in en-US, and en-US is in the list of dictionaries in "spellchecker.dictionary", we'll only enable the en-US dictionary. If there's no match on the element or document, then all of the dictionaries in the pref are used.

Comment 39

[Ben Bucksch (:BenB)]

element or document language ... is ... en-US ..., we'll only enable the en-US dictionary

So, if Gmail or Yahoo or RoundCube document was set to "en-US", that means I can only type English emails? Doesn't that counter the purpose of this bug?

Comment 40

[Alexander Gavrilov [:dartraiden]]

that means I can only type English emails?

You can type in any language, but only English dictionary will be enabled for such pages. You will have to manually enable multiple dictionaries via spellchecker context menu.

Comment 41

[Ben Bucksch (:BenB)]

When I wrote "can type", I meant of course with the spellchecker working in the language, assuming I have the right dictionary installed, without the spellchecker trying to tell me that all my words are wrong and everything has a red underline, which is quite annoying. (Underline for illustration of the effect.)

It would be quite a pity to do all this work in this bug, and then the spellchecker still assumes English, just because Gmail or Yahoo Mail or RoundCube or whatever web app is in English. How would RoundCube know which language I am writing the email in? I write emails (and text in webapps) in multiple languages, so this logic as stated in comment 38 would render the fix here moot in such situations and keep the spellchecker broken for all languages but one.

Comment 42

[Ben Bucksch (:BenB)]

I often even mix multiple languages in the same text, for various reasons. In a multi-language context, this is quite common.

Comment 43

[Alexander Gavrilov [:dartraiden]]

Yes, I wrote about this in https://bugzilla.mozilla.org/show_bug.cgi?id=1073827#c40
See also https://bugzilla.mozilla.org/show_bug.cgi?id=1073827#c33

I think the priority of choosing dictionaries should be discussed in that ticket

Comment 44

[Dan Minor [:dminor]]

The default for a site you haven't visited is to match the language of the document or the edit control. If you change the dictionaries manually, they should be remembered for the site, so this should only be a problem the first time you visit a site.

I think it's worth discussing whether we should change this behaviour when the user has explicitly enabled multiple dictionaries at the same time, but I think that would best be done either in a new bug or maybe one of the existing bugs above.

Comment 45

[daniel]

(In reply to Dan Minor [:dminor] from comment #38)

(In reply to Alexander Gavrilov from comment #36)

Has the about:config → spellchecker.dictionary setting been adapted to support multiple dictionaries? In my tests setting it to something comma-separated like "ru,en-US" does nothing.

We first try to match the element or document language and then use this preference. So if we think the input is in en-US, and en-US is in the list of dictionaries in "spellchecker.dictionary", we'll only enable the en-US dictionary. If there's no match on the element or document, then all of the dictionaries in the pref are used.

The thing people are looking for is something like https://imgur.com/a/2JUXlwr where you set a default set of languages (toggle on/off) that the browser checks when doing spellchecking.