549474 - [kitsune] Spelling suggestions for kitsune searches

Status: RESOLVED FIXED

(support.mozilla.org :: Search, defect)

Description

[James Socol [:jsocol, :james]]

We need to re-implement spelling suggestions on the Python side. I've looked at a few libraries but am having trouble finding anything newer than a few years old. What I have found so far is:

PyEnchant
python-aspell

If we can't find anything good, we'll roll a suggestions package together, but if anyone has suggestions before that, I'm all ears.

Comment 1

[James Socol [:jsocol, :james]]

Now looking into PyHunspell[1], which needs a little coddling and wrapping but uses Hunspell on the back end, which is the same spelling engine used by Firefox.

[1] http://code.google.com/p/pyhunspell/wiki/UsingPyHunspell

Comment 2

[Paul Craciunoiu [:paulc]]

Certainly sounds promising!

Comment 3

[James Socol [:jsocol, :james]]

Paul: r? http://github.com/jsocol/kitsune/compare/development...bug-549474

Run |pip install -r requirements.txt| after pulling this branch.

Note that to actually see suggestions (in English) without bug 549013, I added the following line to my settings_local::

    LANGUAGES = (('en-us', 'English'),)

Then I went to |localhost:8000/en-us/search?q=spelld wrong| and it make a suggestion. (You can't just use 'en', because there's no generic 'en' dictionary, you need en-us, or another en-* I guess.) (Also note that leaving that line in will cause most of the tests to fail. I'll fix them up after bug 549013 lands.)

So this patch adds a few things:

* Two new helpers, registered with the templates: spellcheck(phrase, locale) and phrase|suggestions(locale).
* Several tests of those helpers.
* A DICT_DIR value in settings.py that tells DidYouMean where to look for dictionaries.
* Adds the request preprocessor to TEMPLATE_CONTEXT_PROCESSORS. (Note that this required adding the whole tuple to settings.py.)
* Two new requirements, pyhunspell and didyoumean. 
* Finally, it uses the helpers in the search results template.

Does not support custom word lists yet, I know. Filed bug 550103 to add them (which, in turn, caused me to open issue 1 for didyoumean on github).

Comment 4

[James Socol [:jsocol, :james]]

As Paul noticed, I forgot to mention that you need to install the hunspell headers. On Fedora this is |yum install hunspell-devel|, on Ubuntu it's |apt-get install libhunspell-dev| (or something similar).

Comment 5

[Paul Craciunoiu [:paulc]]

r+ on the branch, last of commits was here:
http://github.com/jsocol/kitsune/tree/5ed781e28172916617669083978ca465d033c8d8

Comment 6

[James Socol [:jsocol, :james]]

http://github.com/jsocol/kitsune/commit/b585a2f9c68f8fb94d7341f5ccd5f05b9b517a92

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

I don't get the "Did you mean: <strong>crashes</strong>" text on http://support-stage-new.mozilla.com/en-US/search?w=3&q=crashs that I see on http://support.mozilla.com/search.php?where=all&locale=en-US&q=crashs.

James: do we need a bug for IT to file dictionaries on Kitsune staging?  Or is it just hooking up that text?