878209 - [osumo] Implement search functionality for offline

Status: RESOLVED FIXED

(support.mozilla.org :: General, defect)

Description

[Shuhao]

This is one of the more difficult tasks. It requires us to create an efficient search that would work well on Firefox OS (the benchmark phone is the unagi).

Here are some notes:

 - IndexedDB does not offer full-text search. We would have to (semi-)implement one ourselves. This search feature needs to be fast and somewhat accurate.
 - There needs to be an index that's precomputed to speed up the search. The current idea is to simply use a reverse index and compute AND operations when searching for multiple words. 
 - The database we are searching over is localized. Languages such as zh-CN have different ways to specify words. 
 - The database we are searching over is not too big. However, it is still relatively non-trivia (especially for searching languages such as zh-CN, which I can already see as not being too fun).
 - While ranking is a complicated subject itself, I think we need a very rudimentary form of ranking in order for the search terms to be useful. 
 - All of the above needs to be fast enough to work on the unagi.

Some ideas:

 - Pregenerate the index on server.
 - Use a reverse index approach. The index will map from word => [documents]
   - multiple keywords implies using multiple queries and finding the intersection of the documents.
     - This may not work well as adjacent words should be more important than just random words.
     - Languages like Chinese can define a word as one or more characters. There are barely any whitespace/punctuations to help us.
   - This reverse index could just search through the summary/title.
     - Should be a relatively good approximation and good speed.
   - The reverse index could be computed via something like TF-IDF, or other ways to find important words in an article.
     - Approach will miss certain phrases searched.

At the end of the day, all we need to have is a search feature that works okay but is fast. I'm currently leaning towards the idea of searching only the title and the summary as well as precompute the index.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

If we're doing nightly-generated bundles, we could do what Sphinx does and generates a big search file. That enables Sphinx docs to search using only JavaScript client side.

It's probably worth looking at Sphinx for ideas especially for a prototype.

Comment 2

[Shuhao]

Sphinx's search sounds like a cool thing to check out. I'll hopefully get to that by the middle of next week!

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Just to clarify, I mean this Sphinx:

    http://sphinx-doc.org/

Not this Sphinx:

    http://sphinxsearch.com/

Comment 4

[Shuhao]

Yeah. I have used the feature and will investigate the source

Comment 5

[Shuhao]

This is fixed. Here's the correct documentations: http://osumo.readthedocs.org/en/latest/offlinesearch.html