1023000 - [meta] significantly improve gloda indexing and search performance

Status: NEW

(MailNews Core :: Database, defect)

Description

[Wayne Mery (:wsmwk)]

Collection of ideas to significantly improve performance of gloda indexing process, and gloda searching process. The bugs are pretty well documented thanks to asuth and others.  Other ideas lifted from distant discussions.

So far...
bug 484646  gloda may want to VACUUM or auto-VACUUM 
bug 551209  Gloda should asynchronously enter/open folders 	
bug 843639  indexing / downloading email with hundreds of email addresses, memory usage rises to ~300MB/sec until it crashes
bug 585429  Significantly speed up global indexing (gloda)
bug 681754  gloda fts3 tokenizer would greatly benefit from stopword support
bug 632791  Gloda indexes one message per minute with high CPU. Excessively sized global-messages-db.sqlite caused by buggered, excessively sized offline store. 

Excluded because the overall perf benefit will not be significant:
bug 475857  gloda could specialize for gmail IMAP
bug 466246  Gloda could be better about reindexing as a result of flag changes

Toss ups for consideration
- Bug 554190  gloda indexer should prioritize indexing for real-time responsiveness over batch indexing
- Sharding has been mentioned in discussions, but does not have a bug report.
- Might updating sqlite would have any significant benefit?
- Do profiles on network drives have unidentified issues?
- Anything which gets IO off main thread (that isn't already)

I think the last significant perf bug fixed was in version 10, Bug 677805 - Bump page size and cache size for Gloda.