365727 - Efficient "Advanced Search" should start short e-mail parts like date or subject before time consuming body search.

Status: NEW

(MailNews Core :: Search, enhancement)

Description

[Marten Lehmann]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0

When I'm using an extended search, which means:

match all for for following:
- body contains "x"
- subject contains "y"
- date is after "01.12.2006"

the the search is not done very efficiently. The application should look for indexed values first, i.e. subject and date, because search can be done very fast here. Then the application would only have to parse those messages for the body, that matched the requirements of the indexed values. Instead, the application is currently checking for all criterias in a row and is only combining these at the end. With thousands of messages in a mailbox it makes a huge difference if the search in the body is done for all messages or just for those after the 01.12.2006 containing an "y" in the subject.


Reproducible: Always

Steps to Reproduce:
Please see above

Comment 1

[David :Bienvenu]

we let the user control the order of search evaluation by the order they define the search criteria...there has been resistance in the past to the idea of having the search code re-order the terms...

Comment 2

[Marten Lehmann]

I cannot see that this changes anything at all. Order doesn't seem to be important. Even if I'm putting the terms like this

match all for for following:
- subject contains "y"
- date is after "01.12.2006"
- body contains "x"

Mozilla Mail searches _all_ message bodies for "x", not only the messages that met the two criteria above.

Comment 3

[David :Bienvenu]

I think that was fixed for 2.0

Comment 4

[Marten Lehmann]

Release 2.0 what? I'm talking about the mail-client within the Mozilla Application suite or Seamonkey, not Thunderbird. I can only see some 1.1.x betas.

Comment 5

[David :Bienvenu]

oh, sorry, it looked like you filed the bug with firefox...anyway, I think Seamonkey 1.1.x betas correspond to TB 2.0 betas.

Comment 6

[Rainer Bielefeld]

With EN-US Seamonkey 2.33.1 (German Language pack)  Gecko/20100101 Build 20150321194901 (Default Theme) on German WIN7 64bit I observe:

a) Test 1
First Search criterion: Date after today
Second Search criterion: Body contains "e"
Search will be done within a fraction of a second (thousands of e-mails in inbox)

b) Test 2
First Search criterion: Body contains "e"
Second Search criterion: Date after today
Search will Take a minute or so (thousands of e-mails in inbox)

Tests show that criteria sort order matters in current seamonkey versions.
I will do a test with TB later.

c) I wonder whether Search function really works with indexes (I doubt). I think search for Summary or date simply is faster because these parts of an e-mail are smaller than body. So it sounds plausible that "AND" combined search should start with fast search for date or so, then more time consuming search for body only for results of first pass. I think there should be some priority order: always first do Date search, then attachment status, then .... and so on.

d) I modified Summary from "Make better software" to something meaningful with falsifiability.

e) Mail News Core until it will have been proved that that works better in TB.

Comment 7

[Rainer Bielefeld]

There are some other "Make Mail Search Faster" reports: <https://bugzilla.mozilla.org/buglist.cgi?cmdtype=runnamed&namedcmd=365727DUPs&list_id=12193690>
Some of them allegedly fixed, see "Bug 166502 - Mail search does a complete boolean evaluation", "Bug 295088 - Filter/Search term evaluator is extremely sub-optimal". I will try to clarify relations later.