586609 - Let Quick Filter use the gloda index for faster filtering

Status: RESOLVED WONTFIX

(Thunderbird :: Search, enhancement)

Description

[IagoSRL]

User-Agent:       Mozilla/5.0 (Windows; Windows NT 5.1; rv:2.0b3) Gecko/20100805 Firefox/4.0b3 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100802 Lightning/1.0b2 Thunderbird/3.1.2

Quick Filter use the gloda index for faster filtering instead normal filtering (when Gloda and indexing is enabled and the folder to be filtered is not being indexed -to avoid errors in the results-)

Reproducible: Always

Comment 1

[Wayne Mery (:wsmwk)]

What do you mean by "not being indexed", and "to avoid errors..."?
And why do you think using gloda will be faster?
And please give more details about which filter options are not working well.

Comment 2

[rsx11m]

Switching the quick-filter search to Gloda would leave users who switched off the global indexer on purpose (e.g., low-resource machines like netbooks) without the means of searching altogether from the main window.

Comment 3

[Wayne Mery (:wsmwk)]

(In reply to comment #2)
> Switching the quick-filter search to Gloda would leave users who switched off
> the global indexer on purpose (e.g., low-resource machines like netbooks)
> without the means of searching altogether from the main window.

in theory, not necessarily/totally. modulo being able to filter message bodies, I would hope gloda is low cost on such machines if it doesn't index all message bodies, i.e. autosync is off.

Comment 4

[IagoSRL]

When Thunderbird 3.0 was released one of the major enhancements was the 'gloda indexing system', I understand that this is faster than the old search system (Ctrl+Shift+F), Is this wrong?
If is faster, why not use it? If gloda indexing is disabled, filtering will work like now (with the current filter system, you don't lose features), but if gloda is enabled, gloda index will be used (maybe some test -to verify that gloda is faster for filtering- are required...)

> -to avoid errors in the results-
I explain this: if you are doing a filter in a folder and gloda system is indexing messages, or the folder index is not up to date, maybe at this time the index is incomplete and use it can produce incomplete results; at this case, TB can choose use the 'classic' system.

Ok, the bug adds complexity to TB filtering system, but if the performance won is high can be interesting. But if is not fast use gloda for filtering, of course, this bug is no necessary.

Comment 5

[Ludovic Hirlimann [:Usul]]

Andrews know if this is worth it or not. Let's ask the one who knows.

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

Apart from body searches, the non-gloda search system is going to be faster than gloda searches for everything quick filter does.  non-gloda search might still win for many body searches since disk IO could end up being roughly linear and prefetching will really work in our favor.

So, no, not worth it on the complexity or performance fronts.

Comment 7

[IagoSRL]

Sorry, let me note another context:

At my work I am in a 'Windows domain' with roaming profiles -this last means, all my profile, including TB personal files, are copy to the server when I close my Windows session and is verified/copied when I open my session, where more files and size means slower-. To avoid this slowness, I move my TB mail folders -imap and local folders- to my personal shared folder in the file-server. I use Local Folders to archive messages and save space in my store at the mail-server. Now, when I filter my folders, this is slow -I understand it, files are in the server..-, but gloda index file is in my computer and use it for search (*and maybe filtering*) is faster -because is a local file-.
We migrated a few months from a MS Exchange/Outlook environment that have not this problems with sizes and is faster.

I know it's a very specific case, and a non interesting professional context.
Sorry again, is well like 'wontfix'

Comment 8

[Andrew Sutherland [:asuth] (he/him)]

When you are looking at a folder and thus able to use quick filter, all of the messages are already in-memory.  The only way a quick filter needs to touch the disk is if you are doing a text filter with 'body' enabled.  It's conceivable that a gloda fulltext search might win in your specific file location, but given the difference in expected I/O patterns and even accounting for network traffic, the file server could still easily win.  Given the very minor chance for a performance win in a limited set of cases and the code complexity entailed, it's not something I would accept a patch for, so this is wontfix.

(But don't worry, we are working on improving performance in these situations, just via other means.)

Comment 9

[IagoSRL]

Thanks Andrew for the details, I will use the 'body' filter only when really I need it.
And I will wait the another performances for these situations ;)

Thanks at all.

Comment 10

[Emily Bowman]

So far, it seems like very little performance improvement has been made for quick filter body searches. It's also non-obvious that selecting body is the reason for the extreme performance degradation; I've lived with it for years without knowing why, until I came across this bug.

Please reopen as something like "Use Gloda for quick filter *body* searches," given that it is hundreds of times faster than the current linear read of the entire mbox file.

The most bizarre part of the perf monitoring that I found is that quick filter first reads the entire file forwards, and then reads it again backwards.