Last Comment Bug 722743 - [PATCH] Use sqlite FTS for faster text queries
: [PATCH] Use sqlite FTS for faster text queries
Status: RESOLVED FIXED
:
Product: Webtools
Classification: Server Software
Component: DXR (show other bugs)
: Trunk
: All All
: -- normal (vote)
: ---
Assigned To: Carlos Garnacho
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2012-01-31 09:16 PST by Carlos Garnacho
Modified: 2012-04-27 02:58 PDT (History)
3 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
0001-dxr-index-Use-FTS4-sqlite-table-to-store-searchable-.patch (4.09 KB, patch)
2012-01-31 09:16 PST, Carlos Garnacho
ehsan: review+
Details | Diff | Splinter Review
second patch, make search.cgi use FTS4 (8.80 KB, patch)
2012-01-31 09:17 PST, Carlos Garnacho
ehsan: review+
Details | Diff | Splinter Review

Description Carlos Garnacho 2012-01-31 09:16:14 PST
Created attachment 593125 [details] [diff] [review]
0001-dxr-index-Use-FTS4-sqlite-table-to-store-searchable-.patch

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.4+ (KHTML, like Gecko) Version/5.0 Safari/535.4+ Epiphany/3.2.1

Steps to reproduce:

I am attaching 2 patches to make DXR use sqlite's FTS4 for searches within the code, this table contains all plain text content from files, which replaces both files_index.txt and index_index.txt. 

FTS4 MATCH queries result in a much lower constant time per-query for text search, with overall times typically quite below files_index.txt searching. It also results in a lower disk usage on large projects as the tokenization payload is incidently lower than storing line numbers in ascii
Comment 1 Carlos Garnacho 2012-01-31 09:17:19 PST
Created attachment 593126 [details] [diff] [review]
second patch, make search.cgi use FTS4
Comment 2 :Ehsan Akhgari 2012-01-31 13:51:10 PST
Comment on attachment 593125 [details] [diff] [review]
0001-dxr-index-Use-FTS4-sqlite-table-to-store-searchable-.patch

Nit: please remove the spaces before open parenthesis.  r=me with that!
Comment 3 :Ehsan Akhgari 2012-02-02 08:06:41 PST
Comment on attachment 593126 [details] [diff] [review]
second patch, make search.cgi use FTS4

Looks great!
Comment 4 Lionel Dricot 2012-04-27 02:58:30 PDT
merged

Note You need to log in before you can comment on or make changes to this bug.