474730 - should search newest first - searching returns messages in bad order, making searches very time-consuming and irrelevant for large databases

Status: RESOLVED FIXED

(Thunderbird :: Search, enhancement)

Description

[Pawel Pohl]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5 FirePHP/0.2.1
Build Identifier: 2.0.0.17

I have migrated to Thunderbird from Outlook because Outlook's search feature was too slow. I have imported a database with 2 years worth of emails. Thunderbird's search is faster, but it starts with the *oldest* messages; that means that for full body searches (ex.: body contains: "new ftp password") I have to wait 5+ minutes to get to newer (more relevant) messages.

I know that I can limit the messages to, say, last 100 days or 400 days; however, I usually don't know how many days ago the relevant message was sent. Besides, that is an ugly and non-elegant solution.

Reproducible: Always

Steps to Reproduce:
1. Get a big email database
2. Start either a simple or advanced search using "body contains".
Actual Results:  
The search takes a long time (which is OK given the data size), result list is updated on-the-fly (which is very good), but oldest results appear first (which is undesired).

Expected Results:  
The results appear starting from most recent messages.

I am doing a local search, not an IMAP search. 

I am marking this bug/feature request as "major", because it makes one of the most used functions (search) unusable in my case.

Comment 2

[Ludovic Hirlimann [:Usul]]

Steps to Reproduce:
1. Get a big email database
2. Start either a simple or advanced search using "body contains".
Actual Results:  
The search takes a long time (which is OK given the data size), result list is
updated on-the-fly (which is very good), but oldest results appear first (which
is undesired).

Expected Results:  
The results appear starting from most recent messages.

Comment 3

[Pawel Pohl]

Ludo(In reply to comment #1)
> 
> *** This bug has been marked as a duplicate of bug 439697 ***

This is a completely different bug. Note that I am not complaining about the ordering of the display list. I am complaining about the order in which the messages are *searched*.

I am very disappointed to see this bug unresolved for so long. I have just come back to this issue after trying to find another message from three weeks ago. I entered the search term, and the search started about 5 minutes ago. I managed to navigate here and find this thread and write this comment, and the search result page is only half-way through 2008. Please, please change the order in which the messages are *searched*. Searching is not an instantaneous procedure, it takes a lot of time, which is why I presume the search results list is updated live.

I marked this as "polish" because, while I am ignorant to the underlying code, I suspect that it will only require a small change in the searching code. Correct me if I am wrong.

Comment 4

[Ludovic Hirlimann [:Usul]]

Search is being completely rewritten in TB 3.0 - the new functionality should be released pretty soon - we will make announces mdat and on blogs - give it a go and tell us if it solves your issue.

Comment 5

[Pawel Pohl]

Thanks, I will make sure to update as soon as Tbird 3 is out!

Comment 8

[Emilis Dambauskas]

I am using Thunderbird 3.1.8 on Ubuntu and this is not solved yet.

I see this bug registered for version 2.0. Should I register a new bug?

Comment 10

[David Hodges]

Still not fixed more than two years later in Thunderbird 8 ! Please please fix this!

Comment 11

[sba]

I am using TB 12 and its still not solved! Please fix this!

Comment 12

[john]

In ver 15, this still executes the search from oldest to newest.   Evil, with large data sets when searching through the body [for text].

May I suggest either a default direction reversed, into Ver 16 or 17, or allow for directional suggestions in the UI

Couldn't be a huge change, here, could it ?

Can anyone pinpoint the code where the search is called . . . I'll take a look, if someone gives me a starting point.

Comment 13

[john]

Still "upside down" search order in Ver 16.0.1

Comment 14

[Magnus Melin [:mkmelin]]

Attachment: proposed fix

Simple one-liner

Comment 15

[Mark H. David]

I would ideally like to see it search in the order of display. E.g., could be by 'from' from z to a, assuming you'd clicked the 'from' header once or twice effect that order. Then you would not need a separate option. In the short run, I'd settle for just searching in date order from newest to oldest.

Comment 16

[john]

Mark raises a good point -- Magnus: can you make the call to [Reverse]EnumerateMessages follow the users ordering....?

might be a bit of complexity if the user's sort order isn't by date?

Comment 17

[Magnus Melin [:mkmelin]]

Yes that would be more complex, and the use cases for it are not that many - then you'd need to know e.g. who it's from and that that who happens to be in a suitable place in the alphabet. And for date we shouldn't do it anyway, since you might very well want to sort the messages in one order, but in most cases what's newest is more important.

Comment 18

[Mark Banner (:standard8)]

Comment on attachment 671123 [details] [diff] [review]
proposed fix

Ok, I don't see any issues with this. r=me.

Comment 19

[Magnus Melin [:mkmelin]]

http://hg.mozilla.org/comm-central/rev/ed60a8f558aa -> FIXED

Comment 20

[Magnus Melin [:mkmelin]]

Backed out since this seems to have caused some oranges -- https://tbpl.mozilla.org/php/getParsedLog.php?id=16737186&tree=Thunderbird-Trunk

http://hg.mozilla.org/comm-central/rev/f0a1dc6af47c

Comment 21

[Magnus Melin [:mkmelin]]

Also to check - https://tbpl.mozilla.org/php/getParsedLog.php?id=16736733&tree=Thunderbird-Trunk&full=1

Comment 22

[Magnus Melin [:mkmelin]]

(In reply to Magnus Melin from comment #21)
> Also to check -
> https://tbpl.mozilla.org/php/getParsedLog.php?id=16736733&tree=Thunderbird-
> Trunk&full=1

This one wasn't related, went away before the backout.

Comment 23

[Magnus Melin [:mkmelin]]

The test failure is always reproducible, but i don't think i see a real world bug (though the refresh isn't doable the same way from the ui afaik). The failure is here: http://mxr.mozilla.org/comm-central/source/mail/base/test/unit/test_viewWrapper_virtualFolder.js#350

Andrew, any idea about this?

Comment 24

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Magnus Melin from comment #23)
> Andrew, any idea about this?

I accuse the reverse enumerator of stopping prematurely, skipping the first header in the box, but the quick perusal I did of the code in question was not enough to make it obvious where the glitch is.  I would suggest looking into the enumerator more.

Comment 25

[Magnus Melin [:mkmelin]]

Attachment: proposed fix, v2

Try looks happy with this (except for the usual oranges) - https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=cf1e60fd4c8a

Comment 26

[Ludovic Hirlimann [:Usul]]

Comment on attachment 724560 [details] [diff] [review]
proposed fix, v2

I'm pretty It's :asuth that you wanted as a reviewer - switching.

Comment 27

[Magnus Melin [:mkmelin]]

Thx, must have misclicked.

Comment 28

[john]

Any chance that :asuth might look see in on this ?

Comment 29

[Magnus Melin [:mkmelin]]

Andrew? ping

Comment 30

[Andrew Sutherland [:asuth] (he/him)]

Comment on attachment 724560 [details] [diff] [review]
proposed fix, v2

r=asuth, tremendously sorry about the delay.

Comment 31

[Magnus Melin [:mkmelin]]

https://hg.mozilla.org/comm-central/rev/d195627d5482 -> FIXED

Comment 32

[john]

Out of curiosity, Magnus, when is v24.0 actually scheduled...?  My up-to-date version is only at 17.0.6, now....

(Or: Is there a reason why such fixes aren't merely inserted into the next release, say, 17.0.7 ?)

I ask, because this patch could be immediately useful.

Comment 33

[john]

(In reply to john ruskin from comment #32)
.... say, 17.0.7 ?)

Or to any next available version, before 24, before 3 months from now?  My vote would be to proffer it to the next release.   

I plead ignorance about how the versioning system works, schedules, and etc., and the system for choosing the version into which a patch is placed.

Comment 34

[Magnus Melin [:mkmelin]]

It's available in nightly builds already if you're in a hurry and willing to accept some rough edges. The next non-developer version is tb24, which will be released in september - https://wiki.mozilla.org/Releases#Next_Thunderbird_Major_Release

These kind of fixes aren't backported to the stable branch (tb 17), as patches can always cause unwanted side effects nobody thought they would.