Last Comment Bug 474730 - should search newest first - searching returns messages in bad order, making searches very time-consuming and irrelevant for large databases
: should search newest first - searching returns messages in bad order, making ...
Status: RESOLVED FIXED
: polish
Product: Thunderbird
Classification: Client Software
Component: Search (show other bugs)
: 2.0
: All All
: -- enhancement with 3 votes (vote)
: Thunderbird 24.0
Assigned To: Magnus Melin
:
:
Mentors:
: 645984 699392 886122 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-01-21 18:25 PST by Pawel Pohl
Modified: 2013-10-22 10:42 PDT (History)
18 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
proposed fix (1.26 KB, patch)
2012-10-13 12:30 PDT, Magnus Melin
standard8: review+
Details | Diff | Splinter Review
proposed fix, v2 (5.54 KB, patch)
2013-03-13 12:37 PDT, Magnus Melin
bugmail: review+
Details | Diff | Splinter Review

Description Pawel Pohl 2009-01-21 18:25:29 PST
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5 FirePHP/0.2.1
Build Identifier: 2.0.0.17

I have migrated to Thunderbird from Outlook because Outlook's search feature was too slow. I have imported a database with 2 years worth of emails. Thunderbird's search is faster, but it starts with the *oldest* messages; that means that for full body searches (ex.: body contains: "new ftp password") I have to wait 5+ minutes to get to newer (more relevant) messages.

I know that I can limit the messages to, say, last 100 days or 400 days; however, I usually don't know how many days ago the relevant message was sent. Besides, that is an ugly and non-elegant solution.

Reproducible: Always

Steps to Reproduce:
1. Get a big email database
2. Start either a simple or advanced search using "body contains".
Actual Results:  
The search takes a long time (which is OK given the data size), result list is updated on-the-fly (which is very good), but oldest results appear first (which is undesired).

Expected Results:  
The results appear starting from most recent messages.

I am doing a local search, not an IMAP search. 

I am marking this bug/feature request as "major", because it makes one of the most used functions (search) unusable in my case.
Comment 1 Ludovic Hirlimann [:Usul] 2009-02-13 00:54:45 PST

*** This bug has been marked as a duplicate of bug 439697 ***
Comment 2 Ludovic Hirlimann [:Usul] 2009-03-06 04:58:05 PST
Steps to Reproduce:
1. Get a big email database
2. Start either a simple or advanced search using "body contains".
Actual Results:  
The search takes a long time (which is OK given the data size), result list is
updated on-the-fly (which is very good), but oldest results appear first (which
is undesired).

Expected Results:  
The results appear starting from most recent messages.
Comment 3 Pawel Pohl 2009-06-09 00:10:22 PDT
Ludo(In reply to comment #1)
> 
> *** This bug has been marked as a duplicate of bug 439697 ***

This is a completely different bug. Note that I am not complaining about the ordering of the display list. I am complaining about the order in which the messages are *searched*.

I am very disappointed to see this bug unresolved for so long. I have just come back to this issue after trying to find another message from three weeks ago. I entered the search term, and the search started about 5 minutes ago. I managed to navigate here and find this thread and write this comment, and the search result page is only half-way through 2008. Please, please change the order in which the messages are *searched*. Searching is not an instantaneous procedure, it takes a lot of time, which is why I presume the search results list is updated live.

I marked this as "polish" because, while I am ignorant to the underlying code, I suspect that it will only require a small change in the searching code. Correct me if I am wrong.
Comment 4 Ludovic Hirlimann [:Usul] 2009-06-09 01:17:19 PDT
Search is being completely rewritten in TB 3.0 - the new functionality should be released pretty soon - we will make announces mdat and on blogs - give it a go and tell us if it solves your issue.
Comment 5 Pawel Pohl 2009-06-09 02:50:19 PDT
Thanks, I will make sure to update as soon as Tbird 3 is out!
Comment 6 [:Aureliano Buendía] 2011-04-05 10:21:53 PDT
*** Bug 645984 has been marked as a duplicate of this bug. ***
Comment 7 [:Aureliano Buendía] 2011-04-05 10:23:21 PDT
*** Bug 647828 has been marked as a duplicate of this bug. ***
Comment 8 Emilis Dambauskas 2011-04-29 02:13:43 PDT
I am using Thunderbird 3.1.8 on Ubuntu and this is not solved yet.

I see this bug registered for version 2.0. Should I register a new bug?
Comment 9 Magnus Melin 2011-11-04 02:34:15 PDT
*** Bug 699392 has been marked as a duplicate of this bug. ***
Comment 10 David Hodges 2011-11-25 02:05:10 PST
Still not fixed more than two years later in Thunderbird 8 ! Please please fix this!
Comment 11 sba 2012-04-26 07:47:03 PDT
I am using TB 12 and its still not solved! Please fix this!
Comment 12 john ruskin 2012-10-13 07:56:55 PDT
In ver 15, this still executes the search from oldest to newest.   Evil, with large data sets when searching through the body [for text].

May I suggest either a default direction reversed, into Ver 16 or 17, or allow for directional suggestions in the UI

Couldn't be a huge change, here, could it ?

Can anyone pinpoint the code where the search is called . . . I'll take a look, if someone gives me a starting point.
Comment 13 john ruskin 2012-10-13 08:34:52 PDT
Still "upside down" search order in Ver 16.0.1
Comment 14 Magnus Melin 2012-10-13 12:30:21 PDT
Created attachment 671123 [details] [diff] [review]
proposed fix

Simple one-liner
Comment 15 Mark H. David 2012-10-13 13:57:14 PDT
I would ideally like to see it search in the order of display. E.g., could be by 'from' from z to a, assuming you'd clicked the 'from' header once or twice effect that order. Then you would not need a separate option. In the short run, I'd settle for just searching in date order from newest to oldest.
Comment 16 john ruskin 2012-10-13 15:19:45 PDT
Mark raises a good point -- Magnus: can you make the call to [Reverse]EnumerateMessages follow the users ordering....?

might be a bit of complexity if the user's sort order isn't by date?
Comment 17 Magnus Melin 2012-10-14 03:27:49 PDT
Yes that would be more complex, and the use cases for it are not that many - then you'd need to know e.g. who it's from and that that who happens to be in a suitable place in the alphabet. And for date we shouldn't do it anyway, since you might very well want to sort the messages in one order, but in most cases what's newest is more important.
Comment 18 Mark Banner (:standard8, limited time in Dec) 2012-11-01 03:12:17 PDT
Comment on attachment 671123 [details] [diff] [review]
proposed fix

Ok, I don't see any issues with this. r=me.
Comment 19 Magnus Melin 2012-11-04 03:41:51 PST
http://hg.mozilla.org/comm-central/rev/ed60a8f558aa -> FIXED
Comment 20 Magnus Melin 2012-11-04 10:59:36 PST
Backed out since this seems to have caused some oranges -- https://tbpl.mozilla.org/php/getParsedLog.php?id=16737186&tree=Thunderbird-Trunk

http://hg.mozilla.org/comm-central/rev/f0a1dc6af47c
Comment 22 Magnus Melin 2012-11-07 11:28:05 PST
(In reply to Magnus Melin from comment #21)
> Also to check -
> https://tbpl.mozilla.org/php/getParsedLog.php?id=16736733&tree=Thunderbird-
> Trunk&full=1

This one wasn't related, went away before the backout.
Comment 23 Magnus Melin 2012-11-07 11:36:57 PST
The test failure is always reproducible, but i don't think i see a real world bug (though the refresh isn't doable the same way from the ui afaik). The failure is here: http://mxr.mozilla.org/comm-central/source/mail/base/test/unit/test_viewWrapper_virtualFolder.js#350

Andrew, any idea about this?
Comment 24 Andrew Sutherland [:asuth] 2012-11-07 13:01:24 PST
(In reply to Magnus Melin from comment #23)
> Andrew, any idea about this?

I accuse the reverse enumerator of stopping prematurely, skipping the first header in the box, but the quick perusal I did of the code in question was not enough to make it obvious where the glitch is.  I would suggest looking into the enumerator more.
Comment 25 Magnus Melin 2013-03-13 12:37:28 PDT
Created attachment 724560 [details] [diff] [review]
proposed fix, v2

Try looks happy with this (except for the usual oranges) - https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=cf1e60fd4c8a
Comment 26 Ludovic Hirlimann [:Usul] 2013-03-14 04:04:12 PDT
Comment on attachment 724560 [details] [diff] [review]
proposed fix, v2

I'm pretty It's :asuth that you wanted as a reviewer - switching.
Comment 27 Magnus Melin 2013-03-14 12:01:46 PDT
Thx, must have misclicked.
Comment 28 john ruskin 2013-05-21 18:42:56 PDT
Any chance that :asuth might look see in on this ?
Comment 29 Magnus Melin 2013-06-04 13:04:07 PDT
Andrew? ping
Comment 30 Andrew Sutherland [:asuth] 2013-06-06 16:53:59 PDT
Comment on attachment 724560 [details] [diff] [review]
proposed fix, v2

r=asuth, tremendously sorry about the delay.
Comment 31 Magnus Melin 2013-06-07 11:16:42 PDT
https://hg.mozilla.org/comm-central/rev/d195627d5482 -> FIXED
Comment 32 john ruskin 2013-06-25 05:57:34 PDT
Out of curiosity, Magnus, when is v24.0 actually scheduled...?  My up-to-date version is only at 17.0.6, now....

(Or: Is there a reason why such fixes aren't merely inserted into the next release, say, 17.0.7 ?)

I ask, because this patch could be immediately useful.
Comment 33 john ruskin 2013-06-25 06:09:01 PDT
(In reply to john ruskin from comment #32)
.... say, 17.0.7 ?)

Or to any next available version, before 24, before 3 months from now?  My vote would be to proffer it to the next release.   

I plead ignorance about how the versioning system works, schedules, and etc., and the system for choosing the version into which a patch is placed.
Comment 34 Magnus Melin 2013-06-25 06:23:08 PDT
It's available in nightly builds already if you're in a hurry and willing to accept some rough edges. The next non-developer version is tb24, which will be released in september - https://wiki.mozilla.org/Releases#Next_Thunderbird_Major_Release

These kind of fixes aren't backported to the stable branch (tb 17), as patches can always cause unwanted side effects nobody thought they would.
Comment 35 Jim Porter (:squib) 2013-10-22 10:42:21 PDT
*** Bug 886122 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.