342913 - [meta] Tracking: Add full-text indexing to bookmark and/or history searches

Status: NEW

(Firefox :: Bookmarks & History, enhancement, P3)

Description

[Pam Greene]

Now that SQLite, the back-end storage for Places, is going to support
full-text indexing (http://www.sqlite.org/cvstrac/wiki?p=FullTextIndex), we can
significantly enhance Firefox's search capabilities by searching more
than titles and URLs.

Among the design questions to be considered are what to index, how and where to display the results, and what options to support.

This bug is intended to serve as both a tracking bug for individual portions of the work and a discussion point (in addition to mozilla.dev.apps.firefox) for design ideas.

Comment 1

[Nickolay_Ponomarev]

Note http://skrul.com/blog/projects/xpclucene/ , a CLucene-based solution, which is less elegant than using sqlite, but still mentioning.

Comment 2

[O. Atsushi (Torisugari)]

I'm happy to see mozilla starts hacking natural language processing.

Then we'll really go with SQLite? I'm not too sure yet, but I'm afraid
that full text indexing in SQLite does not work nicely with CJK languages.

中文と日本語では、単語が空白で区切られていません。
BothinChineseandJapanese, wordsarenotseparatedwithwhitespaces.

To slove this probelm, Lucene uses "Tokenizer"s for specific languages.
http://lucene.apache.org/java/docs/api/org/apache/lucene/analysis/class-use/Analyzer.html
http://lucene.apache.org/java/docs/api/org/apache/lucene/analysis/Tokenizer.html
However, CJKAnalyzer sucks. And I don't believe SQLite will implement
anything better than that, in a short period.

On the other hand, I guess commercial search engines such as Google
and Yahoo are doing some kind of linguistic approach, especially
Morphology, though I don't know exactly what they are doing.  
http://en.wikipedia.org/wiki/Morpheme
http://en.wikipedia.org/wiki/Conditional_random_field

Unless we'll be able to tokenize input properly, this feature
will never reach the product-level quality. Anyway, the backend
is not just a wrapper of a SQL database. At least we should
support lang-dep-tokenizer switching.

Comment 3

[Pam Greene]

I doubt any CJK tokenizer will be included with SQLite by default, but the text-indexing system is designed to allow developers to plug in custom functions.  The trick will be finding (or writing) good ones.

Comment 5

[Gervase Markham [:gerv]]

Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h".

In Thunderbird 3.0b, you do that as follows:
Tools | Message Filters
Make sure the correct account is selected. Click "New"
Conditions: Body   contains   places-to-b-and-h
Change the action to "Delete Message".
Select "Manually Run" from the dropdown at the top.
Click OK.

Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter.

Gerv

Comment 7

[Nicolas Barbulesco]

I want the words I type in the address bar to match the title of the pages in my history. I believe that Firefox does that now.
I don't want them to match the text content à la Chrome, because it is too heavy, slows down everything.
But being able to search, in the History panel and window, the text content of the pages would be great !

Comment 9

[Jonathan Watt [:jwatt]]

Mass bug change to replace various 'parity' whiteboard flags with the new canonical keywords. (See bug 1443764 comment 13.)

Comment 10

[ddavidwang]

(In reply to Nicolas Barbulesco from comment #7)

I want the words I type in the address bar to match the title of the pages
in my history. I believe that Firefox does that now.
I don't want them to match the text content à la Chrome, because it is too
heavy, slows down everything.
But being able to search, in the History panel and window, the text content
of the pages would be great !

Yes, it'd be preferred that this is optional and non-default if it is implemented.