1557974 - Default path for hunspell dictionaries

Status: RESOLVED WONTFIX

(Core :: Spelling checker, enhancement, P3)

Description

[Erik Persson]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36

Steps to reproduce:

Tried to install (ie put symbolic links) hunspell dictionaries (.dic .aff) in <installdir>/dictionaries/

It does not work in v68.0b8 either.

For long I was able to add hunspell dictionaries by putting them into the <installdir>/dictionaries directory. It seems like this very nice default behavior was removed in Firefox version 62. Before the current version it was easy to add dictionaries and use the same dictionaries as in the rest of the system.
Directories to search for hunspell dictionaries can however be added by using the DICPATH environment variable. Based on the source code of extensions/spellcheck/hunspell/glue/mozHunspellDirProvider.cpp the former default behavior could however be added with an else clause to the if (dicEnv) {...}

Actual results:

Nothing

Expected results:

I should be able to use the dictionaries.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:smaug, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 2

[Olli Pettay [:smaug][[e-mail]]]

The initial comment isn't super clear. So DICPATH works but using links to add new dictionaries doesn't work?

Comment 3

[Erik Persson]

In earlier releases ( < version 62) of Firefox <install_dir>/dictionaries could be used for Hunspell dictionaries without setting any environment variables. This was the default behavior. This behavior is described on multiple places on "the net", for example on mozillazine.org and in at least one answer on support.mozilla.org. The use of DICPATH in Firefox is however harder to find. In recent releases (from version 62) Firefox does no longer by default search for Hunspell dictionaries in <install_dir>/dictionaries/.

The addition of DICPATH (from ~ version 55) is welcome and I have not found any bugs related to the use of DICPATH. However, as far as I can see a small change to the source code would restore the former default behavior probably without any major side effects. This would restore a functionality that has been in Firefox for long, which is expected and which is described on multiple places on the internet. The use of DICPATH is not well documented. In fact I searched for long, and I suspected that the functionality of using external Hunspell dictionaries was removed since I couldn't find anything related to DICPATH (or spellchecker.dictionary_path ). The solution to using external Hunspell dictionaries is mostly described as "put the files into <install_dir>/dictionaries/", and when you do it does no longer work.

Thus the changes from version 62 breaks an established behavior, and it may be easy to restore the former functionality.

Comment 4

[Olli Pettay [:smaug][[e-mail]]]

DICPATH was added in bug 1310835.

The dictionary handling was changed in bug 1457321.
I doubt we want to bring that old behavior back, especially if DICPATH works.
But Kris may have an opinion here.

Comment 5

[Erik Persson]

Ok. Sad.

However, I don't think bug 1457321 applies. As far as I can see the changes in bug 1457321 deals with requiring all dictionary files to be in directories and as unpacked files. A return to this is not at all what I suggest.

The actual overhead of what I suggest, that is to check if <install_dir>/dictionary/ exists if DICPATH is not set, seems to be easy to fix in the current code and on fairly old hardware with spinning disks on linux kernel 4.4 it takes about 5 us (microseconds) to check for the existence of a non existent directory. Thus the overhead compared to the current situation is negligible, and it is not comparable to the io overhead described in the bug.

Comment 6

[Kris Maglione [:kmag]]

Yes, this is the expected behavior. We don't want to have to scan directories at startup, and we don't want to load unpacked dictionaries at all if we don't have to. If a dictionary is being installed specifically for Firefox, it should be installed as an extension.

The legacy directory scanning behavior only exists for the sake of Linux distros which point Firefox at their global dictionary directory.

Comment 7

[Erik Persson]

Yes, and no one has asked for any mandatory scanning of any directory or any mandatory reading of any unpacked file, but just an about 5 microsecond (that is 5 millionth of a second) check for the existence of a directory to avoid breaking an established behavior!
(And IF the directory exists THEN scan it etc, but in general it should not exist and thus the overhead would be minimal. Ie for users who don't use the feature the overhead is close to 0 and for users who use it, the overhead is the same as when setting DICPATH to an existing directory.)

The only change to the source code would be something like:

if (! dicEnv ) {
  dicEnv = "<install_dir>/dictionaries";
}

after line 267 in mozHunspell.cpp. I assume there is a standard way to get the install directory.

I have used the "dictionaries in <install_dir>/dictionaries" feature not only on linux but also on windows using the same dictionaries (multiple installations of firefox on different platforms used the same files on the local net - very convinient). I think FF on the osx machines use it as well. Thus I don't think it is only linux users who use this, but I guess we aren't that many, and there will be even less of us using DICPATH since it is poorly documented.

But I get it, for some reason - the about 5 millionth of a second can hardly be the explanation behind breaking an established behavior - you don't want this and there is another way to solve the problem.

Other than that, thank you for a very nice browser (even though it got a tiny bit less nice).

Comment 8

[Kris Maglione [:kmag]]

That 5μs number is pure supposition. In practice, it will vary considerably.

Regardless, this is not something we're going so support. As I already said, if you want to install dictionaries globally specifically for Firefox, you need to install them as extensions. If you want to point Firefox at some other sort of global dictionary directory, you need to set the environment variable.

Comment 9

[Erik Persson]

No, the 5 us is NOT pure supposition since, if you read what I have written, I tested how long it would take to try to open a non existent directory. This however depends on the relevant file system tables being loaded into cache or kernel space. Since firefox is loaded from the very same directory containing the "directories" directory, it is VERY LIKELY that the corresponding file system tables are already loaded into memory. If they are not, it will instead take in the order of microseconds (compare for example with the seek times of spinning disks). The time it takes is in the same order as for other system calls, for example to malloc, since it is unlikely that any IO is ever done if the directory does not exist.

Thus, this is not even a question about IO. It is just a system call since the relevant file system tables very likely already are in memory. I assume you are a programmer, and you know this.
The time does not even vary very much. I have then tested it on 5 other machines and the time it takes to try to open a non existent directory varies between 2 us and up to 25 us (linux, freebsd and and an old core 2 duo macbook, all with spinning disks but when the filesystem tables are in memory, as would be the case with firefox). There will, ofcourse, be more code, but unless the code makes many system calls the rest of the code should not take much more than some 50 000 cycles = 25 us on a 2 GHz machine.

Thus the proposed but rejected changes would carry only a negligible overhead, and if it takes 5 us or 50 us does not matter. In fact even if it would vary 1000%, it would not matter. Thus rejecting it on grounds of overhead is NOT a valid argument. (Neither is, for example, that it would require large changes to the code.)
It is a simple change that would restore earlier behavior and it comes with almost no overhead.

However, you, ofcourse, decide, and you don't want it and you can reject it for whatever reasons you like, but don't say it is because of overhead. You just don't want the functionality, even if it breaks an established behavior.