Open Bug 1437111 Opened 6 years ago Updated 2 years ago

spelling dictionary repeatedly switches from US to UK English

Categories

(Core :: Spelling checker, defect, P3)

Product:
Core
Component:
Spelling checker
Version:
58 Branch
Platform:
x86_64
Linux
Type:
defect

Tracking

()

People

(Reporter: hangfirew8, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(3 files)

Add-ons Manager.png
43.76 KB, image/png
Details
Screenshot of Add-On Tab
17.46 KB, image/png
Details
spellcheck.png - screenshot of about:config
16.59 KB, image/png
Details 
User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0
Build ID: 20180205205729

Steps to reproduce:

Type in any text field
Notice red underlines prompting UK spelling
Right-Click Languages, find that English (United Kingdom) is once again selected
Change to English (United States)
Keep working
repeat above, find it back on UK again.

Mint "Language Settings", "Select your default session language" set to: "English, United States" but System Language is set to "C" so bash sorts properly. "Number of languages installed: 1"

Firefox->Configure->Languages set to, in this order:
English/United States [en-us] 
English [en]




Actual results:

keeps reverting spelling dictionary to UK from US English


Expected results:

Stay on selected language please
Blocks: spell-lang-change
Component: Untriaged → Spelling checker
OS: Unspecified → Linux
Product: Firefox → Core
Hardware: Unspecified → x86_64
How many dictionaries do you have installed? Why not just remove the UK one if you don't want it?
Uninstallation is a good workaround. However since Firefox supports multiple dictionaries, we should probably figure out why it has this behaviour, and if it's desired.

@hangfirew8 Do you find that it switches back on a given domain, or is incorrect only on new domains until you change it?

I recently hit this problem and haven't fully characterised it yet, but right now I have English (Australia) on Facebook (correct since LANG="en_AU.UTF-8") and English (United States) on mozilla.org and arbitrary newly visited domains. Gnome language is English (Australia), Firefox languages are en-us and us (however that shouldn't matter).

I seem to recall the same issue being present years ago.
When this happened many years ago, I remember removing some dictionaries as a workaround, but I wouldn't call it a "good" workaround, for several reasons:

1) This workaround *conflicts* with other directives.  This should be pretty obvious.  I didn't install the other dictionaries; they satisfy other dependencies (or recommendations).  I'm not responsible for optimizing my dictionary list *for FF.*  Available dictionaries are a function of other things, not how FF does spell checking.

2) I cannot control the dictionaries that get installed, at least not at this level.  This is the default set.

3) A dictionary matching my locale is installed.  I've done my part.  Why is FF (still) ignoring my direct, explicit directive regarding language preferences?  This is why this frustrating.  I know there will be bugs, but this problem was solved decades ago.
Ack!  Sorry, i tried to kill that post, but bugzilla wouldn't forget.  Never mind! I've explained all of that before on the other bug.  Sorry sorry sorry.
(In reply to Jorg K (GMT+1) from comment #1)
> How many dictionaries do you have installed? 

The only dictionaries installed were listed in my original post, and do not include anything labeled "uk":

English/United States [en-us] 
English [en]

> Why not just remove the UK one if you don't want it?
Because a.) there is no "uk" dictionary, and b.) I wasn't sure if the en-us can stand alone.

(In reply to Paul from comment #2)
> @hangfirew8 Do you find that it switches back on a given domain, or is
> incorrect only on new domains until you change it?
It will happen on any website I spend time on, and it will happen repeatedly if I spend enough time on any. Also when FF is restarted it will default back to UK.

> I seem to recall the same issue being present years ago.
As do I. It went away for a while, now it's back.
(In reply to Porcelain Mouse from comment #4)
> Ack!  Sorry, i tried to kill that post, but bugzilla wouldn't forget.  Never
> mind! I've explained all of that before on the other bug.  Sorry sorry sorry.
No problem from me, I feel the same way. 

Could you like the "other bug" please?
nm I see it's the "blocks" https://bugzilla.mozilla.org/show_bug.cgi?id=1073827
There are two different configurations in Firefox:
  Tools > Options, Language
and
  Tools > Add-ons, Dictionaries.
If there is no UK English or AU English dictionary, then Firefox cannot possibly spellcheck in UK or AU English.

If you have both US and UK/AU English installed, the dictionary that will be used depends on the site you visit and whether you've chosen a specific language for that site before (which is remembered in the so-called content preferences).

The algorithm is described here and the behaviour hasn't changed since 2015:
https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#830

You can always check the dictionary that is used on the context menu:
Right-click into a text field and check what you have under "Languages".
Based on that algorithm, it seems the problem is that spellchecker.dictionary never gets set.

It's not defined by default in spellchecker.dictionary. If I create it with a value of "en-AU", the default correctly becomes English (Australia). But changing the spell checker language via the context menu, even on a domain never visited before, doesn't change the value of spellchecker.dictionary.

When spellchecker.dictionary isn't set, step 2 of the algorithm doesn't preference the user's last choice, but rather falls back to the "(almost) at random" behaviour. Which explains why both @hangfirew8 and were getting different wrong dictionaries, though one of us wanted the system default and the other did not.

Where is spellchecker.dictionary supposed to be getting set, and why isn't it happening? Does the test at line 681 now fail due to e10s?

https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#681
Wow, that change was made after 2015 :-(
https://hg.mozilla.org/mozilla-central/rev/3328527cb7d5#l1.14

Nicholas, why was it a good idea to change the dictionary selection mechanism?
Flags: needinfo?(n.nethercote)
(In reply to Jorg K (GMT+1) from comment #8)
> There are two different configurations in Firefox:
>   Tools > Options, Language
> and
>   Tools > Add-ons, Dictionaries.
> If there is no UK English or AU English dictionary, then Firefox cannot
> possibly spellcheck in UK or AU English.

Wait.  Hold the phone.  There isn't any special section in Add-ons called "Dictionaries."  Do you mean the set of add-ons that you can find with the search "dictionaries"?  Because, in that case, I have no extras like that installed.  Also, I think it's "languages" not "dictionaries".  Furthemore, I don't see any add-ons to help select the default language.

You have missed one of the sources of dictionaries: system dictionaries.  On Linux, there is a long standing system of application-agnostic spelling dictionaries.  Firefox uses these, also.  So, why doesn't it have a way to select the default?

Once again, I point out that Firefox is not the only software on my computer and needs to respect the rest of installed applications.  None of the other programs are confused about my language, only Firefox.

It all makes sense now: what changed recently was more dictionaries got installed because a program I installed requested it.
Attached image Add-ons Manager.png — 
Screenshot of the Add-ons Manager on Windows showing the Dictionaries.

To my knowledge Firefox only uses its own dictionaries, that come either bundled with a language pack or bundled with Firefox, like US English. From vague memory it's possible that on Linux extra language packs appear under Add-ons.

Have you checked the "Languages" item in the context/right-click menu?

Anyway, we've discovered a problem here: The dictionary selection mechanism might not work due to e10s changes, see comment #10.
> Nicholas, why was it a good idea to change the dictionary selection
> mechanism?

Each process has its own copy of the prefs table. And there is a clear invariant: pref values can only be changed in the parent process. Notifications of those changes then get forwarded on to all content processes so they can update their copies.

Prior to my change, this invariant was not being strictly enforced, and this editor code was violating it. This means a content process would change the value of its own copy of the spellchecker.dictionary pref, while the copies of the same pref in all other processes were unchanged. This is a dangerous inconsistency, because e.g. the value shown in about:config and the value written to the prefs.js file (if it's changed) comes from the parent process.

I apologize for the regression; I should have checked with an Editor peer about the possible effects of this. Having said that, the old behaviour was untenable, so we need to find a different way to do this. Perhaps the child can send a message to the parent requesting a dictionary change?

Masayuki, do you have ideas about how to fix this?
Flags: needinfo?(n.nethercote) → needinfo?(masayuki)
(In reply to Jorg K (GMT+1) from comment #12)
> Created attachment 8951746 [details]
> Add-ons Manager.png
> 
> Screenshot of the Add-ons Manager on Windows showing the Dictionaries.

I confess, I haven't noticed that before but I hardly ever use Windows.  Could it be that devs have not use Linux version?  Because that doesn't exist on that platform.

> To my knowledge Firefox only uses its own dictionaries, that come either
> bundled with a language pack or bundled with Firefox, like US English. From
> vague memory it's possible that on Linux extra language packs appear under
> Add-ons.

Incorrect.  I have never use the add-on "dictionaries," so I cannot say if they work or not on Linux; they seem to be install-able and I assume they do work.  But, the ones I'm complaining about do not come from that.  The problem of spell check was solved long before FF.  I think it is great that this preexisting solution is usable by FF.  That should be a good thing, but it seems that this capability has only made things worse, somehow.  Since Windows never had a generic spell checker available for all apps, perhaps this additional solution was developed for that platform, but it has made it impossible for users and devs to communicate clearly.

> Have you checked the "Languages" item in the context/right-click menu?

You mean in free text fields?  Yes, they're are many languages there, more than 10, but "US-English (Malawi)" seem to the "default" because it's the one that is selected for all web pages.

> Anyway, we've discovered a problem here: The dictionary selection mechanism
> might not work due to e10s changes, see comment #10.

Cool.  I'm guessing that will fix the problem...sort of.  Why is the default wrong in the first place?  It doesn't need to be wrong because I'm telling FF what language I want to use when I start it.

I don't know why everyone keeps ignoring my comment regarding locale.  Please don't be offended if you know how locale works, but I'll explain in case some people don't.  You store your preferred language in the the LANG environment variable.  The value is already in the *exact* form you want for choosing dictionary files from the install set of available languages.  You check this value, and that could be the default language.  Done.  Why doesn't FF use this value?  That's what it's there for.  No matter how many languages are installed, this tells *all* programs which one I want to use.  For example, if a program crashes before loading some configuration that would tell it what language I like, *this* is the language used to print the error.  This is part of how "internationalization" works.  I know *some* part of FF *is* using this value because FF is internationalized.  Why not use it in the spell checker?  Every single Linux computer in the world has this value set.  This *is* how this works, at least on Linux.  You can do your own thing, for sure, but why, when someone already solved the problem and the code exists?

I'm just explaining why it frustrating to Linux users to see this bug.  If you say that that just will not work in the code, okay, fine.  I believe you; you are the expert, not me.  You don't have to explain any further.  But, just know that Linux users are being very explicit about what they want and FF is ignoring it.  It's not that FF doesn't have a setting that I want; it has the setting I want, I've set it correctly, and it's not working.
With respect, as a long time Linux user: please calm down.

This bug is not related to the problem you have described. You may want to refer your concerns, *in a constructive fashion*, to the parent bug 1073827.

Ever since Mozilla adopted XUL, the modus operandi has been to integrate functionality directly into the browser, so as to provide a similar experience to users no matter which operating system they happen to be using. Aside from OS-specific functionality such as the PRIMARY selection, we don't get an exemption from this just because our OS has built-in spell-checking or some other fancy stuff.

Please understand that we're all on the same team here, and if you can come up with an improvement to the spell checker selection algorithm that solves bug 1073827, you will score a goal for the team. But please discuss it with us (under the appropriate bug) like you're on our side.
(In reply to Nicholas Nethercote [:njn] from comment #13)
> ..., so we need to find a different way to do this. Perhaps the child
> can send a message to the parent requesting a dictionary change?
If you look for blame around 
https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#830
you will find that I landed a few bugs in 2015 to straighten out dictionary selection behaviour since there was a whole swag of complaints in bug 1073827.

The logic is still somewhat twisted, but it relies on a user choice being stored in spellchecker.dictionary and used later. So anything that restores the spirit of the original algorithm, like sending a message to the parent, would be fine IMHO.

===

Note: Sadly we have a second thread happening in this bug in comment #11, comment #12, comment #14 and comment #15. I'm not sure how to untangle that. Maybe Porcelain Mouse could file another bug for Linux re. the UI issues.

Just briefly: It seems that that on Linux you have a "global" English pack provided by your distribution, with many flavours, including "US-English (Malawi)". The dictionary selection mechanism documented at
https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#830
is still in force even in this scenario. It is likely that the bug we're discussing here, user choice not stored, effects the behaviour. However, since this bug here was reported for FF 58 and the behaviour has changed in FF 59 in bug 1416622, it looks like we're discussing two different issue here. Or are you on FF 59 beta?

Finally, bug 1073827 is a meta-bug, please don't discuss anything there. File a new bug and make it block bug 1073827.
With respect, and as a long time Linux user, I'm pretty calm and clearly on the same side.  I thought that was coming through.

I'm on FF 58; I'll file another bug.  No problem.  Thanks, all.
> I'm on FF 58; I'll file another bug.  No problem.  Thanks, all.

Thank you! Having separate bugs for separate issues is very helpful.
Thank you all for identifying the origin of this bug.

On my 58.0.2 Linux FF (I think FF has updated since I filed this bug) I see Add-ons Manager->Languages (not Dictionaries) and indeed it has English (GB) Language Pack and English (South Africa) Language Pack. I never installed these, so they must come with either Firefox or Mint packaging.

If there is a language guessing algorithm, it's not very good. I get reverted to the UK dictionary while interacting with US web sites.
Add-Ons->Languages is a curious beast, as (on my Linux FF anyway) there is no mechanism (that I can find) for adding a Language Pack. I have to find a text entry field to get a context menu and choose Languages->Add Dictionaries... to do that.
(In reply to hangfirew8 from comment #20)
> ..., so they must come with either Firefox or Mint packaging.
I believe this is the case.

> If there is a language guessing algorithm, it's not very good. I get
> reverted to the UK dictionary while interacting with US web sites.
You need to understand the algorithm:
https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#830
If the website requests "en" and you have en-GB, en-ZA and en-US, you will get any "en" at random. If however you chose en-US for that site, that will be remembered and also for other sites which request "en", this will now be used.
(In reply to Nicholas Nethercote [:njn] from comment #13)
> > Nicholas, why was it a good idea to change the dictionary selection
> > mechanism?
> 
> Each process has its own copy of the prefs table. And there is a clear
> invariant: pref values can only be changed in the parent process.
> Notifications of those changes then get forwarded on to all content
> processes so they can update their copies.
> 
> Prior to my change, this invariant was not being strictly enforced, and this
> editor code was violating it. This means a content process would change the
> value of its own copy of the spellchecker.dictionary pref, while the copies
> of the same pref in all other processes were unchanged. This is a dangerous
> inconsistency, because e.g. the value shown in about:config and the value
> written to the prefs.js file (if it's changed) comes from the parent process.
> 
> I apologize for the regression; I should have checked with an Editor peer
> about the possible effects of this. Having said that, the old behaviour was
> untenable, so we need to find a different way to do this. Perhaps the child
> can send a message to the parent requesting a dictionary change?
> 
> Masayuki, do you have ideas about how to fix this?

I'm not familiar with spellchecker, though, I read the code briefly. Currently, I understand the behavior around the "spellchecker.dictionary" pref is, EditorSpellCheck::SetCurrentDictionary() is always called in the process which is in the editor and it's called when user selects a dictionary from context menu explicitly or dictionaries (https://searchfox.org/mozilla-central/rev/5536f71c3833018c4f4e2c73f37eae635aab63ff/toolkit/modules/InlineSpellCheckerContent.jsm#120) are added into context menu (https://searchfox.org/mozilla-central/rev/5536f71c3833018c4f4e2c73f37eae635aab63ff/toolkit/modules/InlineSpellChecker.jsm#227). So, if editor is in content, the pref won't be modified. On the other hand, nobody observes this pref change. So, this pref is used only when it's referred directly. It's referred only by EditorSpellCheck::SetFallbackDictionary() which is called only by EditorSpellCheck::DictionaryFetched(). It's called when nsIInlineSpellChecker.updateCurrentDictionary() is called. It's typically called when editor gets focus (https://searchfox.org/mozilla-central/rev/5536f71c3833018c4f4e2c73f37eae635aab63ff/editor/libeditor/EditorBase.cpp#5555) or inline spellchecker is enabled explicitly (https://searchfox.org/mozilla-central/rev/5536f71c3833018c4f4e2c73f37eae635aab63ff/editor/libeditor/EditorBase.cpp#1417). So, if user changes dictionary in chrome, the pref is typically modified and it'll be copied to every remote process (I assume so). Then, when moving focus to editor in content, dictionary could be changed accidentally. So, perhaps, as you said, EditorSpellCheck should post a message to parent process to modify the pref (perhaps, EditorSpellCheck -> mEditor->AsEditorBase()->GetWidget()->NewMethod() -> TabChild::SendNewMessage() -> TabParent::RecvNewMessage() -> Preferences::SetString()). However, there is short delay until pref in content is actually modified. If it causes some problem, we need to make EditorSpellCheck store the pref value with a member variable, but perhaps, it's not problem since currently, the pref won't be modified if it's in content but we don't know some bugs caused by this.
Flags: needinfo?(masayuki)
[ Triage 2017/02/20: P3 ]
Priority: -- → P3
> If however you chose en-US for that site, that will be remembered and also for other sites which request "en", this will now be used.

Is there another way to choose other than the Context Menu->Languages->(pick one)? Because I just did that on this site, moved from English (United Kingdom) to English (United States), brought up a new tab with Facebook, and it was defaulting to UK there.

Also I have had Add-ons->Languages both language packs GB and AZ "disabled" and this behavior of defaulting to the UK spelling dictionary continues.
Since the code has been reviewed and the behavior know, can this be changed from UNCONFIRMED?
Really? Still unconfirmed? I have the other libraries DISABLED and it still defaults/switches to them, the bug is known...
Nicholas, any update here? This
https://hg.mozilla.org/mozilla-central/rev/3328527cb7d5#l1.14
really changes the dictionary selection mechanism.
Flags: needinfo?(n.nethercote)
Comment 13 still holds. The editor code needs to be changed, perhaps in the way described in comment 23. I don't know anything about editor code so I'm not in a position to do that myself.
Flags: needinfo?(n.nethercote)
Masayuki-san, could you or someone from your team get this fixed? I understand that not setting spellchecker.dictionary any more impacts the language selection and inconveniences some users who operate with multiple dictionaries.
Flags: needinfo?(masayuki)
Nobody works on spellchecker, additionally, nobody familiar with spellchecker as far as I know. Makoto-san might know something, but I don't know he is available for this bug.

(summaries of the cause are: comment 8, comment 13 and comment 23)
Flags: needinfo?(masayuki)
(In reply to Masayuki Nakano [:masayuki] (JST, +0900) (away: 4/28 - 5/6) from comment #31)
> Nobody works on spellchecker, additionally, nobody familiar with
> spellchecker as far as I know. Makoto-san might know something, but I don't
> know he is available for this bug.
I used to work on the spellchecker, see:
https://hg.mozilla.org/mozilla-central/log/03afd4febf0b/editor/libeditor/nsEditor.cpp
https://hg.mozilla.org/mozilla-central/log/08cd638ca00d/editor/composer/nsEditorSpellCheck.cpp
https://hg.mozilla.org/mozilla-central/log/59b6a838f0f2/extensions/spellcheck/hunspell/src/mozHunspell.cpp
https://hg.mozilla.org/mozilla-central/log/59b6a838f0f2/editor/libeditor/nsEditorEventListener.cpp
That was back in 2015 when I cleaned up an awful mess and closed most of the bugs blocking bug 1073827.

I also established a clear algorithm for dictionary selection here:
https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#830

Now sadly someone came along in the context of E10S and destroyed what had worked well for two years (2015-2017). So, IMHO, is there is someone to go in and break things, thus "working" on the code, then there should be someone to go in and fix those regressions.

If you read the algorithm, having the correct value in spellchecker.dictionary is important.

You don't need to much knowledge of spellchecking to write some messaging mechanism to get the preference value stored correctly.

Jet, I'm a bit disappointed that the work I invested in 2015 is disintegrating since no one feels responsible to fix regressions. There is a community of non-en-US English speakers using more than one en-* dictionary and they are once again puzzled by FF's behaviour, a situation I cleaned up previously in 2015. Can you please find a resource to get this fixed.
Flags: needinfo?(bugs)
Although I cannot reproduce this, dictionary selection is stored to ContentPrefService2 (sqlite database), not prefs since we move to e10s.
And for eEditorMailMask, I guess that we should add new preference since spellchecker.dictionary is a kind of localization.
(In reply to Makoto Kato [:m_kato] from comment #33)
> Although I cannot reproduce this, dictionary selection is stored to
> ContentPrefService2 (sqlite database), not prefs since we move to e10s.
Well, no. There are various strategies to get the spell-check dictionary that should be used, please read here:
https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#830
It's also set as a fallback
https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#674
for the next site which doesn't have content preference. This part is now sadly disabled.

(In reply to Makoto Kato [:m_kato] from comment #34)
> And for eEditorMailMask, I guess that we should add new preference since
> spellchecker.dictionary is a kind of localization.
I don't understand that comment. BTW, Thunderbird is not affected, since it always gets the applicable language from the document's root element (option 2, "language of website", or in TB's case, language of document):
https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#749

That allows multiple compositions in different languages. So I want write two e-mail in English and Spanish at the same time and each one is checked in the document's language.

eEditorMailMask has the effect that manually chosen languages are not stored in spellchecker.dictionary:
https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#650

To reproduce this bug, you'd have to visit a website which is "en". Select "en-AU". Then visit another website which is "en". That should now also use "en-AU" since "en-AU" was previously stored in spellchecker.dictionary. If the value is not set, it can happen that en-US or en-GB are used for the second "en" website at random (see description of algorithm). Of course you need those three dictionaries to test it. It's very tricky stuff, so it's sad, that the working algorithm got destroyed. The fix is simply to communicate the choice of language to whichever part of the system which can write to spellchecker.dictionary. As simple as that.
No.  so since editor will work on child process, so we should store dictionary selection to content prefs, not system's preference.  Current design is bad for e10s.

And there is a lack of UI test for this case.  I don't know why this test isn't added on e10s support.
It's funny that you talk about "design" here, when there was no design. Let's look at this in the historic context. In the olden days, there were *no* content preferences. So the spelling was based on the website you were visiting ("lang" attribute" or other), and in absence of that, the preference spellchecker.dictionary. If you chose a dictionary, that was stored in that preference. Later on, bug 678842 came along and stored the language "per site" in the context preferences. spellchecker.dictionary was also still set and used if a) no language was provided by the web page and b) no language preference was previously stored. All this predates e10s.

So the idea behind the language selection for spell-check is this:
1) If the user visited the site before and chose a language, that will be used by the next visit.
   When the user sets a language, we also remember this choice in the global preference
   spellchecker.dictionary and we'll see why we do that in point 3).
2) When the user comes to a site (or text entry field) that uses "en-GB", that dictionary will be
   used if present.
3) If the user comes to a site (or text entry field) that uses simply "en", the system doesn't
   know which dictionary to use if the user has en-US and en-GB which is frequently the case for
   people who use an en-US FF, but install an en-GB dictionary additionally.
   Now the preference comes into play: If the user previously set "en-GB" for some other site
   we assume that they prefer "en-GB" over "en-US" and therefore use "en-GB" on any site
   what doesn't have content preferences stored and that only specifies "en".

Since spellchecker.dictionary is not exposed in the UI (it is in TB), it doesn't really matter where you store the "general" preference of the user when it comes to dictionary selection. It could be stored in some non-site-specific content preference, but that somewhat contradicts the concept of content preferences which define preferences for "content", that is per site.

So summarising: You need to provide a mechanism by which the user can pre-select a certain dictionary if no content preferences are stored and if the site doesn't mandate a certain dictionary.

Why are you opposed to storing that in a preference? Could it be that if content process 1 would be allowed to set the preference via some messaging method, that content process 2 woudn't get notified?
"design" word might not be good for this, but it meat that we might have to remove storing spellchecker.dictionary prefs in editor when we added e10s support.  User confuses the selection for spellchecker.dictionary and content prefs (and localization default) since there will be no way to set spellchecker lang from any setting UI (include about:config) when using content pref.

So, I think that we shouldn't save spellchecker.dictionary in editor code when user select dictionary.  It should be stored to content prefs.  But as default (HTML content has no lang etc),  we might read spellchecker.dictionary.  If user wants to change default lang, they change it by spellchecker.dictionary from about:config.

1. no content prefs, it should respect content lang for dictionary.
   a. content is en-US, it should use en-US, then localization default (spellchecker.dictionary) if lang code is matched, then any en-*, then localization default (spellchecker.dictionary), system lang
   b. content is en-GB, it should use en-GB, then localization default (spellchecker.dictionary) if lang code is matched, then any en-*, then ...
   c. content is en, it should use localization default (spellchecker.dictionary) if if lang code is matched, then any en-*., then ...
2. if there is content prefs, dictionary lang should use it
   a. (user remove dictionary, then 1.)
3. no lang and no content prefs, spellchecker.dictionary is used (but we don't save it to spellchecker.dictionary).
If I understand it correctly, what you're writing in comment #38 is basically what I was going to suggest.

Basically, we leave everything as it is, with the exception, not to change spellchecker.dictionary when the user selects a language for a site. So basically we just remove the code that isn't run anyway any more:
https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#674-686

That leaves the problem of setting spellchecker.dictionary in the first place. It's not in the UI. So maybe we should add a visible option somewhere: "Fallback spellcheck dictionary".

Reporter, does your problem get solved if you set your preferred dictionary in spellchecker.dictionary via about:config?

Also, let's get Zibi on board here. Zibi, good morning! You know that in a text input box, FF does spell checking. There is an intricate algorithm at
https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#830
to select a spellcheck dictionary. Basically it remember the user's choice for a particular site (content prefs), then it tries to derive the language from the website content ("lang" attribute, etc.), then there is a long list of fallbacks. The first fallback comes from pref spellchecker.dictionary.

Now, in the past, that fallback pref was set every time the user selected a language for a particular site, with the following idea in mind: If the guy/girl knows to write in Polish, we will offer them Polish if we have no better idea. Sadly, with e10s, we can't set that preference from a content child process. So my idea is to expose the preference in the UI. Sadly we can't automatically set it to the app locale since FF isn't shipped with a dictionary for most locales. Thoughts?
Flags: needinfo?(hangfirew8)
Flags: needinfo?(gandalf)
Seems like there are two separate topics here:

1) The UI for per-site selection of dictionaries.

Users are notoriously bad at making decisions out of context. Asking them to go to Prefs to select their dictionary for lemonde.fr is suboptimal compared to selecting the dictionary right as they type, in the textbox.

So I think I'd suggest adding an IPC API for setting per-site dictionary in the parent process and calling it from the child process.

If the user wants a pref-per-site, then letting them make the decision right in-context, seems like the best UX.

Then, in Preferences we could expose some UI to view all the per-site options (much like permissions exceptions subdialogs) and let the user manage their exceptions there.

2) Heuristics of guessing the right dictionary selection

My gut feeling is that we should use some selection of:
 * web page lang attribute
 * domain guessing (lemonde.fr -> fr)
 * content language guessing [0]
 * accepted-headers requested locales
 * LocaleService::RequestedLocales

to construct `requestedLocales` for language negotiation [1].

[0] https://searchfox.org/mozilla-central/source/browser/components/translation/LanguageDetector.jsm
[1] https://searchfox.org/mozilla-central/source/intl/locale/mozILocaleService.idl#127
Flags: needinfo?(gandalf)
(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #40)
> Users are notoriously bad at making decisions out of context. Asking them to
> go to Prefs to select their dictionary for lemonde.fr is suboptimal compared
> to selecting the dictionary right as they type, in the textbox.
That's how it works now. Right-click right here, Languages > English (UK), will set the language for BMO to English (UK) and that will be permanently remembered as a content preference. Are you missing something?

Point two seems to opt for removing the preference, right? My suggestion was to expose the preference in the UI.

As for your heuristics: domain guessing doesn't buy you anything since many domains are .com .org .net and many sites are multi-language.
> Point two seems to opt for removing the preference, right?

No, sorry. Point 2 is about picking up the default. Per-site overrides make sense and I think we should keep them. I'd just prefer to keep them in-context rather than out-of-context.

> As for your heuristics: domain guessing doesn't buy you anything since many domains are .com .org .net and many sites are multi-language.

Sure, we'd probably want to use some white-list to check if the top-level-domain matches any of the known language-codes.
I don't see a spellchecker.dictionary in about:config.
I only see a services.sync.prefs.sync.spellchecker.dictionary , and it is a boolean. It is set to True. I have changed it to False. However Sync, while enabled on my primary browser, syncs with no other browser. I also unchecked Preferences under Sync. Will need a little time to see if this fixes anything.
> That's how it works now. Right-click right here, Languages > English (UK),
> will set the language for BMO to English (UK) and that will be permanently
> remembered as a content preference. Are you missing something?

It the selection is "permanently remembered", which does my FF keep switching from US to UK? Often within one session?
Flags: needinfo?(hangfirew8)
I set spelling dictionary to English (United Sates), exited FF, start, navigated to this page and I'm back to UK.
I see, FF have removed spellchecker.dictionary as a visible preferences. You can add it by right-clicking, New > String, spellchecker.dictionary, en-US.

(In reply to hangfirew8 from comment #45)
> I set spelling dictionary to English (United Sates), exited FF, start,
> navigated to this page and I'm back to UK.
Please clarify this. How did you *set* the dictionary to en-US? Via the context (right-click) menu in a text entry field? Then this is stored as a content preference for that site only. After restart, what does "I'm back to UK" mean? On the same site? On another site?

As a workaround, you should simply uninstall the undesired dictionary, but it might be that that's not possible on Linux since the do some funny stuff packaging a bazillion en-* dictionaries together.

The way it should behave is this: When you select en-US while on a specific site, that choice should be remembered for the next visit. If you visit an English site for the first time and the site doesn't set a specific language, then the value from spellchecker.dictionary should be used.

If the choice for a site doesn't stick, the maybe your content preferences database is corrupt. Try deleting content-prefs.sqlite from your profile. Or open the file with an SQLite editor to see what's going on.
Jorg K,

Thank you for your thoughtful response. I'm sorry it took a while to get back to you.

Yes, I set the dictionary via the context (right-click) menu in a text entry field.

"I'm back to UK" means on that site, and every other site.

On my current distro, English langauages are packaged in firefox-locale-en, which includes:

/usr/share/doc/firefox-locale-en
/usr/share/doc/firefox-locale-en/changelog.gz
/usr/share/doc/firefox-locale-en/copyright
/usr/lib/firefox-addons
/usr/lib/firefox-addons/extensions
/usr/lib/firefox-addons/extensions/langpack-en-ZA@firefox.mozilla.org.xpi
/usr/lib/firefox-addons/extensions/langpack-en-GB@firefox.mozilla.org.xpi

Hmm US english is not in there. As a workaround I might be able to remove that package.

Next I will try:
Create spellchecker.dictionary, and set it to en-US
content-pres.sqlite
Remove firefox-locale-en package.

Thanks again, I'll be back in a bit.
Just setting the string spellchecker.dictionary has an immediate positive effect, on a restart my text entry field context menu now default to English (United States).
I tried sqlitebrowser on content-prefs.sqlite and it appeared intact but empty.

I exited FF, deleted the the file and restarted, and made an arbitrary config change (home page). The db browser still showed that file as empty. Spelling dictionary still set to US.

I Reset about:config spellchecker.dictionary, exited FF and started again. Config item spellchecker.dictionary is missing, as expected. Spelling dictionary in text entry fields now defaults to UK.

I recreated about:config spellchecker.dictionary and set it to en-US, reloaded this page, and my dictionary is now set to US. No restart was required for the setting to take effect.

So, I do not believe a corrupt sqlite db is the issue.

I do have a robust workaround, simply set spellchecker.dictionary to en-US.

Thank you for your help. I do believe there is a bug here to be fixed, but at least I can stop fighting switching dictionaries. I will try to stop in, in case you need more testing.
Well, content-prefs.sqlite is a little difficult to navigate, but it shouldn't be empty.

In table "settings" I have id=3, name=spellcheck.lang.

Then in table "prefs" I need to find rows with settingID==3. I have one, id=42 and value=de-DE. Now looking at table "groups" where id=42 I see name=www.ebay-kleinanzeigen.de.

In other words, I visited www.ebay-kleinanzeigen.de once, a German site, and set the dictionary to de-DE. And that will be applied every time I visit this site again.
Just a note. Had to reset FF for other reasons. Bug returned.
Reset about:config spellchecker.dictionary to en-us.

IMHO This bug should be marked CONFIRMED.
Another note Bug exists from 58 through 61.0.1.
Since this bug has been reproduced, can we move it on from UNCONFIRMED please?
Status: UNCONFIRMED → NEW
Ever confirmed: true
Clearly ni on Jet as he no longer works at Mozilla.
Flags: needinfo?(bugs)

comment #55 is test patch.

(In reply to Jorg K (GMT+2) from comment #56)

Shouldn't the code go here somewhere:
https://searchfox.org/mozilla-central/rev/05a22d864814cb1e4352faa4004e1f975c7d2eb9/editor/spellchecker/EditorSpellCheck.cpp#719-891

I think that this issue is "en-*" installed, (almost) at random. case. So if content is 'en' and user locale is 'en-US', we should use 'en-US' instead of 'en-GB'. Right? I will file new issue for it.

It's a long time ago that I looked at the code EditorSpellCheck.cpp. From memory, if the site requires "en", we will select any "en-*" dictionary at random, but also look at the last choice in spellchecker.dictionary. Sadly setting that pref went broken in the e10s effort. I really think we should restore that, so if the user once chose en-US that choice should be used for all "en" websites.

Looking at the application locale, something that wasn't possible years ago, might be another selection criterion. But looking at the pref is more important. Say an Australian user has an en-US version, visits an "en" website and selects the en-AU dictionary they installed. Previously that choice was remembered in spellchecker.dictionary. Now we can't write to the pref in the content process any more. So when the user now visits another "en" website, they will no longer get a random "en" dictionary, but "en-US" all the time. I can't see that this is an improvement. Or am I missing something? That dictionary selection code at the reference I gave in comment #57 was finely tuned and in fact solved all cases dependent on bug 1073827.

Is that something we could use Locale service::NegotiateLanguages for? (Main filtering algo, not the manual selection which has to go through IPC to parent)

(In reply to Zibi Braniecki [:zbraniecki][:gandalf] from comment #59)

Is that something we could use Locale service::NegotiateLanguages for? (Main filtering algo, not the manual selection which has to go through IPC to parent)

Yes. But we have to replace all locale parameters with nsCString. Current is nsString...

Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: