spelling dictionary repeatedly switches from US to UK English

UNCONFIRMED
Unassigned

Status

()

Core
Spelling checker
P3
normal
UNCONFIRMED
14 days ago
3 days ago

People

(Reporter: hangfirew8, Unassigned)

Tracking

(Blocks: 1 bug)

58 Branch
x86_64
Linux
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

14 days ago
User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0
Build ID: 20180205205729

Steps to reproduce:

Type in any text field
Notice red underlines prompting UK spelling
Right-Click Languages, find that English (United Kingdom) is once again selected
Change to English (United States)
Keep working
repeat above, find it back on UK again.

Mint "Language Settings", "Select your default session language" set to: "English, United States" but System Language is set to "C" so bash sorts properly. "Number of languages installed: 1"

Firefox->Configure->Languages set to, in this order:
English/United States [en-us] 
English [en]




Actual results:

keeps reverting spelling dictionary to UK from US English


Expected results:

Stay on selected language please

Updated

13 days ago
Blocks: 1073827
Component: Untriaged → Spelling checker
OS: Unspecified → Linux
Product: Firefox → Core
Hardware: Unspecified → x86_64

Comment 1

13 days ago
How many dictionaries do you have installed? Why not just remove the UK one if you don't want it?

Comment 2

13 days ago
Uninstallation is a good workaround. However since Firefox supports multiple dictionaries, we should probably figure out why it has this behaviour, and if it's desired.

@hangfirew8 Do you find that it switches back on a given domain, or is incorrect only on new domains until you change it?

I recently hit this problem and haven't fully characterised it yet, but right now I have English (Australia) on Facebook (correct since LANG="en_AU.UTF-8") and English (United States) on mozilla.org and arbitrary newly visited domains. Gnome language is English (Australia), Firefox languages are en-us and us (however that shouldn't matter).

I seem to recall the same issue being present years ago.

Comment 3

13 days ago
When this happened many years ago, I remember removing some dictionaries as a workaround, but I wouldn't call it a "good" workaround, for several reasons:

1) This workaround *conflicts* with other directives.  This should be pretty obvious.  I didn't install the other dictionaries; they satisfy other dependencies (or recommendations).  I'm not responsible for optimizing my dictionary list *for FF.*  Available dictionaries are a function of other things, not how FF does spell checking.

2) I cannot control the dictionaries that get installed, at least not at this level.  This is the default set.

3) A dictionary matching my locale is installed.  I've done my part.  Why is FF (still) ignoring my direct, explicit directive regarding language preferences?  This is why this frustrating.  I know there will be bugs, but this problem was solved decades ago.

Comment 4

13 days ago
Ack!  Sorry, i tried to kill that post, but bugzilla wouldn't forget.  Never mind! I've explained all of that before on the other bug.  Sorry sorry sorry.
(Reporter)

Comment 5

7 days ago
(In reply to Jorg K (GMT+1) from comment #1)
> How many dictionaries do you have installed? 

The only dictionaries installed were listed in my original post, and do not include anything labeled "uk":

English/United States [en-us] 
English [en]

> Why not just remove the UK one if you don't want it?
Because a.) there is no "uk" dictionary, and b.) I wasn't sure if the en-us can stand alone.

(In reply to Paul from comment #2)
> @hangfirew8 Do you find that it switches back on a given domain, or is
> incorrect only on new domains until you change it?
It will happen on any website I spend time on, and it will happen repeatedly if I spend enough time on any. Also when FF is restarted it will default back to UK.

> I seem to recall the same issue being present years ago.
As do I. It went away for a while, now it's back.
(Reporter)

Comment 6

7 days ago
(In reply to Porcelain Mouse from comment #4)
> Ack!  Sorry, i tried to kill that post, but bugzilla wouldn't forget.  Never
> mind! I've explained all of that before on the other bug.  Sorry sorry sorry.
No problem from me, I feel the same way. 

Could you like the "other bug" please?
(Reporter)

Comment 7

7 days ago
nm I see it's the "blocks" https://bugzilla.mozilla.org/show_bug.cgi?id=1073827

Comment 8

7 days ago
There are two different configurations in Firefox:
  Tools > Options, Language
and
  Tools > Add-ons, Dictionaries.
If there is no UK English or AU English dictionary, then Firefox cannot possibly spellcheck in UK or AU English.

If you have both US and UK/AU English installed, the dictionary that will be used depends on the site you visit and whether you've chosen a specific language for that site before (which is remembered in the so-called content preferences).

The algorithm is described here and the behaviour hasn't changed since 2015:
https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#830

You can always check the dictionary that is used on the context menu:
Right-click into a text field and check what you have under "Languages".

Comment 9

7 days ago
Based on that algorithm, it seems the problem is that spellchecker.dictionary never gets set.

It's not defined by default in spellchecker.dictionary. If I create it with a value of "en-AU", the default correctly becomes English (Australia). But changing the spell checker language via the context menu, even on a domain never visited before, doesn't change the value of spellchecker.dictionary.

When spellchecker.dictionary isn't set, step 2 of the algorithm doesn't preference the user's last choice, but rather falls back to the "(almost) at random" behaviour. Which explains why both @hangfirew8 and were getting different wrong dictionaries, though one of us wanted the system default and the other did not.

Where is spellchecker.dictionary supposed to be getting set, and why isn't it happening? Does the test at line 681 now fail due to e10s?

https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#681

Comment 10

7 days ago
Wow, that change was made after 2015 :-(
https://hg.mozilla.org/mozilla-central/rev/3328527cb7d5#l1.14

Nicholas, why was it a good idea to change the dictionary selection mechanism?
Flags: needinfo?(n.nethercote)

Comment 11

7 days ago
(In reply to Jorg K (GMT+1) from comment #8)
> There are two different configurations in Firefox:
>   Tools > Options, Language
> and
>   Tools > Add-ons, Dictionaries.
> If there is no UK English or AU English dictionary, then Firefox cannot
> possibly spellcheck in UK or AU English.

Wait.  Hold the phone.  There isn't any special section in Add-ons called "Dictionaries."  Do you mean the set of add-ons that you can find with the search "dictionaries"?  Because, in that case, I have no extras like that installed.  Also, I think it's "languages" not "dictionaries".  Furthemore, I don't see any add-ons to help select the default language.

You have missed one of the sources of dictionaries: system dictionaries.  On Linux, there is a long standing system of application-agnostic spelling dictionaries.  Firefox uses these, also.  So, why doesn't it have a way to select the default?

Once again, I point out that Firefox is not the only software on my computer and needs to respect the rest of installed applications.  None of the other programs are confused about my language, only Firefox.

It all makes sense now: what changed recently was more dictionaries got installed because a program I installed requested it.

Comment 12

7 days ago
Created attachment 8951746 [details]
Add-ons Manager.png

Screenshot of the Add-ons Manager on Windows showing the Dictionaries.

To my knowledge Firefox only uses its own dictionaries, that come either bundled with a language pack or bundled with Firefox, like US English. From vague memory it's possible that on Linux extra language packs appear under Add-ons.

Have you checked the "Languages" item in the context/right-click menu?

Anyway, we've discovered a problem here: The dictionary selection mechanism might not work due to e10s changes, see comment #10.
> Nicholas, why was it a good idea to change the dictionary selection
> mechanism?

Each process has its own copy of the prefs table. And there is a clear invariant: pref values can only be changed in the parent process. Notifications of those changes then get forwarded on to all content processes so they can update their copies.

Prior to my change, this invariant was not being strictly enforced, and this editor code was violating it. This means a content process would change the value of its own copy of the spellchecker.dictionary pref, while the copies of the same pref in all other processes were unchanged. This is a dangerous inconsistency, because e.g. the value shown in about:config and the value written to the prefs.js file (if it's changed) comes from the parent process.

I apologize for the regression; I should have checked with an Editor peer about the possible effects of this. Having said that, the old behaviour was untenable, so we need to find a different way to do this. Perhaps the child can send a message to the parent requesting a dictionary change?

Masayuki, do you have ideas about how to fix this?
Flags: needinfo?(n.nethercote) → needinfo?(masayuki)

Comment 14

6 days ago
Created attachment 8951838 [details]
Screenshot of Add-On Tab

Comment 15

6 days ago
(In reply to Jorg K (GMT+1) from comment #12)
> Created attachment 8951746 [details]
> Add-ons Manager.png
> 
> Screenshot of the Add-ons Manager on Windows showing the Dictionaries.

I confess, I haven't noticed that before but I hardly ever use Windows.  Could it be that devs have not use Linux version?  Because that doesn't exist on that platform.

> To my knowledge Firefox only uses its own dictionaries, that come either
> bundled with a language pack or bundled with Firefox, like US English. From
> vague memory it's possible that on Linux extra language packs appear under
> Add-ons.

Incorrect.  I have never use the add-on "dictionaries," so I cannot say if they work or not on Linux; they seem to be install-able and I assume they do work.  But, the ones I'm complaining about do not come from that.  The problem of spell check was solved long before FF.  I think it is great that this preexisting solution is usable by FF.  That should be a good thing, but it seems that this capability has only made things worse, somehow.  Since Windows never had a generic spell checker available for all apps, perhaps this additional solution was developed for that platform, but it has made it impossible for users and devs to communicate clearly.

> Have you checked the "Languages" item in the context/right-click menu?

You mean in free text fields?  Yes, they're are many languages there, more than 10, but "US-English (Malawi)" seem to the "default" because it's the one that is selected for all web pages.

> Anyway, we've discovered a problem here: The dictionary selection mechanism
> might not work due to e10s changes, see comment #10.

Cool.  I'm guessing that will fix the problem...sort of.  Why is the default wrong in the first place?  It doesn't need to be wrong because I'm telling FF what language I want to use when I start it.

I don't know why everyone keeps ignoring my comment regarding locale.  Please don't be offended if you know how locale works, but I'll explain in case some people don't.  You store your preferred language in the the LANG environment variable.  The value is already in the *exact* form you want for choosing dictionary files from the install set of available languages.  You check this value, and that could be the default language.  Done.  Why doesn't FF use this value?  That's what it's there for.  No matter how many languages are installed, this tells *all* programs which one I want to use.  For example, if a program crashes before loading some configuration that would tell it what language I like, *this* is the language used to print the error.  This is part of how "internationalization" works.  I know *some* part of FF *is* using this value because FF is internationalized.  Why not use it in the spell checker?  Every single Linux computer in the world has this value set.  This *is* how this works, at least on Linux.  You can do your own thing, for sure, but why, when someone already solved the problem and the code exists?

I'm just explaining why it frustrating to Linux users to see this bug.  If you say that that just will not work in the code, okay, fine.  I believe you; you are the expert, not me.  You don't have to explain any further.  But, just know that Linux users are being very explicit about what they want and FF is ignoring it.  It's not that FF doesn't have a setting that I want; it has the setting I want, I've set it correctly, and it's not working.

Comment 16

6 days ago
With respect, as a long time Linux user: please calm down.

This bug is not related to the problem you have described. You may want to refer your concerns, *in a constructive fashion*, to the parent bug 1073827.

Ever since Mozilla adopted XUL, the modus operandi has been to integrate functionality directly into the browser, so as to provide a similar experience to users no matter which operating system they happen to be using. Aside from OS-specific functionality such as the PRIMARY selection, we don't get an exemption from this just because our OS has built-in spell-checking or some other fancy stuff.

Please understand that we're all on the same team here, and if you can come up with an improvement to the spell checker selection algorithm that solves bug 1073827, you will score a goal for the team. But please discuss it with us (under the appropriate bug) like you're on our side.

Comment 17

6 days ago
(In reply to Nicholas Nethercote [:njn] from comment #13)
> ..., so we need to find a different way to do this. Perhaps the child
> can send a message to the parent requesting a dictionary change?
If you look for blame around 
https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#830
you will find that I landed a few bugs in 2015 to straighten out dictionary selection behaviour since there was a whole swag of complaints in bug 1073827.

The logic is still somewhat twisted, but it relies on a user choice being stored in spellchecker.dictionary and used later. So anything that restores the spirit of the original algorithm, like sending a message to the parent, would be fine IMHO.

===

Note: Sadly we have a second thread happening in this bug in comment #11, comment #12, comment #14 and comment #15. I'm not sure how to untangle that. Maybe Porcelain Mouse could file another bug for Linux re. the UI issues.

Just briefly: It seems that that on Linux you have a "global" English pack provided by your distribution, with many flavours, including "US-English (Malawi)". The dictionary selection mechanism documented at
https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#830
is still in force even in this scenario. It is likely that the bug we're discussing here, user choice not stored, effects the behaviour. However, since this bug here was reported for FF 58 and the behaviour has changed in FF 59 in bug 1416622, it looks like we're discussing two different issue here. Or are you on FF 59 beta?

Finally, bug 1073827 is a meta-bug, please don't discuss anything there. File a new bug and make it block bug 1073827.

Comment 18

6 days ago
With respect, and as a long time Linux user, I'm pretty calm and clearly on the same side.  I thought that was coming through.

I'm on FF 58; I'll file another bug.  No problem.  Thanks, all.
> I'm on FF 58; I'll file another bug.  No problem.  Thanks, all.

Thank you! Having separate bugs for separate issues is very helpful.
(Reporter)

Comment 20

5 days ago
Thank you all for identifying the origin of this bug.

On my 58.0.2 Linux FF (I think FF has updated since I filed this bug) I see Add-ons Manager->Languages (not Dictionaries) and indeed it has English (GB) Language Pack and English (South Africa) Language Pack. I never installed these, so they must come with either Firefox or Mint packaging.

If there is a language guessing algorithm, it's not very good. I get reverted to the UK dictionary while interacting with US web sites.
(Reporter)

Comment 21

5 days ago
Add-Ons->Languages is a curious beast, as (on my Linux FF anyway) there is no mechanism (that I can find) for adding a Language Pack. I have to find a text entry field to get a context menu and choose Languages->Add Dictionaries... to do that.

Comment 22

5 days ago
(In reply to hangfirew8 from comment #20)
> ..., so they must come with either Firefox or Mint packaging.
I believe this is the case.

> If there is a language guessing algorithm, it's not very good. I get
> reverted to the UK dictionary while interacting with US web sites.
You need to understand the algorithm:
https://searchfox.org/mozilla-central/rev/74b7ffee403c7ffd05b8b476c411cbf11d134eb9/editor/composer/EditorSpellCheck.cpp#830
If the website requests "en" and you have en-GB, en-ZA and en-US, you will get any "en" at random. If however you chose en-US for that site, that will be remembered and also for other sites which request "en", this will now be used.
(In reply to Nicholas Nethercote [:njn] from comment #13)
> > Nicholas, why was it a good idea to change the dictionary selection
> > mechanism?
> 
> Each process has its own copy of the prefs table. And there is a clear
> invariant: pref values can only be changed in the parent process.
> Notifications of those changes then get forwarded on to all content
> processes so they can update their copies.
> 
> Prior to my change, this invariant was not being strictly enforced, and this
> editor code was violating it. This means a content process would change the
> value of its own copy of the spellchecker.dictionary pref, while the copies
> of the same pref in all other processes were unchanged. This is a dangerous
> inconsistency, because e.g. the value shown in about:config and the value
> written to the prefs.js file (if it's changed) comes from the parent process.
> 
> I apologize for the regression; I should have checked with an Editor peer
> about the possible effects of this. Having said that, the old behaviour was
> untenable, so we need to find a different way to do this. Perhaps the child
> can send a message to the parent requesting a dictionary change?
> 
> Masayuki, do you have ideas about how to fix this?

I'm not familiar with spellchecker, though, I read the code briefly. Currently, I understand the behavior around the "spellchecker.dictionary" pref is, EditorSpellCheck::SetCurrentDictionary() is always called in the process which is in the editor and it's called when user selects a dictionary from context menu explicitly or dictionaries (https://searchfox.org/mozilla-central/rev/5536f71c3833018c4f4e2c73f37eae635aab63ff/toolkit/modules/InlineSpellCheckerContent.jsm#120) are added into context menu (https://searchfox.org/mozilla-central/rev/5536f71c3833018c4f4e2c73f37eae635aab63ff/toolkit/modules/InlineSpellChecker.jsm#227). So, if editor is in content, the pref won't be modified. On the other hand, nobody observes this pref change. So, this pref is used only when it's referred directly. It's referred only by EditorSpellCheck::SetFallbackDictionary() which is called only by EditorSpellCheck::DictionaryFetched(). It's called when nsIInlineSpellChecker.updateCurrentDictionary() is called. It's typically called when editor gets focus (https://searchfox.org/mozilla-central/rev/5536f71c3833018c4f4e2c73f37eae635aab63ff/editor/libeditor/EditorBase.cpp#5555) or inline spellchecker is enabled explicitly (https://searchfox.org/mozilla-central/rev/5536f71c3833018c4f4e2c73f37eae635aab63ff/editor/libeditor/EditorBase.cpp#1417). So, if user changes dictionary in chrome, the pref is typically modified and it'll be copied to every remote process (I assume so). Then, when moving focus to editor in content, dictionary could be changed accidentally. So, perhaps, as you said, EditorSpellCheck should post a message to parent process to modify the pref (perhaps, EditorSpellCheck -> mEditor->AsEditorBase()->GetWidget()->NewMethod() -> TabChild::SendNewMessage() -> TabParent::RecvNewMessage() -> Preferences::SetString()). However, there is short delay until pref in content is actually modified. If it causes some problem, we need to make EditorSpellCheck store the pref value with a member variable, but perhaps, it's not problem since currently, the pref won't be modified if it's in content but we don't know some bugs caused by this.
Flags: needinfo?(masayuki)
[ Triage 2017/02/20: P3 ]
Priority: -- → P3
You need to log in before you can comment on or make changes to this bug.