489711 - quicksearch hangs thunderbird when 'sort by' is threaded and folder is large(3200 messages)

Status: RESOLVED FIXED

(Thunderbird :: Mail Window Front End, defect)

Description

[Bob Noseworthy]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8 (.NET CLR 3.5.30729)
Build Identifier: 2.0.0.21

My setup uses IMAP downloaded for offline use, with an inbox of ~3200 messages. 
Problem is only seen when 'sort by' is threaded and returned search result is large.   
using the quick search window, searches that return many results results in ui hanging - Vista reports "Not responding" and thunderbird must be killed. 

 if the search is by sender and returns only 55 matches (as one example), everything is fine.   
  If the search is by sender and should return ~224 matches, then the ui hangs.
 if the search is by subject and returns ~308 matches everything is fine.  
 if the search is by subject and returns >>308 (perhaps 500+) matches the ui hangs.  

status bar text says "searching..." but if a message is selected to view the ui hangs and never recovers.

Possibly the factor is the length of threads, or nested threads?  rather than number of returned results.    most threads are only 3-8 messages deep, and some are nested one level.  

Problem has existed for sometime pre-2.0.0.21,  though I've typically avoided it by not using the threaded view.  

Reproducible: Always

Steps to Reproduce:
1.Open thunderbird,  ensure view->sortby->threaded is selected (all else defaults)
2.Inbox with >3200 messages, some threaded
3.Select either a subject or sender search (others likely suffer problem as well)
4.Execute search and wait briefly (2-4sec)
5.Attempt to select one of the resulting messages turned up by the search
Actual Results:  
If the result is... too large?  too nested??  then the message cannot be selected, the title bar indicates "Not Responding" and the status bar continues to report "searching..." but does not recover.  Eventually thunderbird must be killed.

Expected Results:  
Search result should return messages and while search is ongoing, messages should be selectable/openable,  or even if waiting till search is complete before trying to select messages, search should complete within 5min (doesn't happen)  

I have deleted my inbox.msf and rebuilt it - no change.  I have compacted my folders - no change.  I've uninstalled possible culprits (gmailui,  gds (google desktop search)) no change.

Comment 1

[Bob Noseworthy]

Possibly related to: https://bugzilla.mozilla.org/show_bug.cgi?id=478822
and less helpfully to https://bugzilla.mozilla.org/show_bug.cgi?id=370100

Comment 2

[WADA:World Anti-bad-Duping Agency]

Thread related issue after getting search result from IMAP server?
Or IMAP search related issue such as "too many results" or "timeout due to long search time"?
Or both?

Quick search with "Threaded View" only issue?
No problem if "Search" in context menu of the IMAP folder?
How about Virtual Folder(Saved Search folder), with/without "Threaded View", with/without "Search Online". 

> User-Agent: (snip) Windows NT 5.1; (snip)
Bob Noseworthy, get IMAP log with timestamp and check IMAP level flow first. 
> Getting NSPR log                     : See Bug 402793 Comment #1
> Adding timestmap to log by DebugView : See Bug 402793 Comment #6
> IMAP command/response                : http://tools.ietf.org/html/rfc3501

Comment 3

[Bob Noseworthy]

Problem occurs even when offline,  so I can't see this being tied to the IMAP server.  

I see NO probelm if I search in the context of my IMAP Inbox folder (right click and 'search')  and perform a search that consistently hangs the 'quicksearch' (when in "threaded view")  - but if I do the same search in the quicksearch box, the ui hangs  (if I disable "threaded view", everything works fine) 

interesting - if I make a saved search folder for a search that routinely hangs from the quicksearch,  the search folder works fine (as it defaults to non-threaded view),  but with the search folder selected,  if I switch the view to threaded,  the ui hangs again (exact same behavior as from the quicksearch)

so it seems to be tied to the search results list and the creation of the threaded view.

other ideas for gathering debug data?

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

Thank you for your willingness to gather debug data!

Please try and reproduce the problem with Thunderbird 3 beta 2 or a recent nightly.  The threading logic for the message views was overhauled for Thunderbird 3, and general view enhancements were made; I would expect the bug no longer exists.

http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-trunk/

A saved search on a single folder uses the exact same code as a quick search on a single folder, so it's not surprising that you are seeing what you are seeing.

Comment 5

[WADA:World Anti-bad-Duping Agency]

Bob Noseworthy, download wi32.zip build from next site.  
> http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-trunk/
You can test latest-trunk by UNZIP only. Please don't forget to test with new profile, in order to avoid corruption of your daily use profile. And don't forget to set "Leave Messages on Server" if you test with POP3.

Comment 6

[WADA:World Anti-bad-Duping Agency]

Oh, Andrew Sutherland(who is the owner of bug 452221) has come.
Andrew, we are better to check issues such as next again with latest-trunk?
  - Issue of Bug 452221
  - Rebuild Index of local mail folder takes long, if all mails has same subject.
  - Change from Non-Threaded-View to Threaded-View takes long, if same subject.

Comment 7

[Andrew Sutherland [:asuth] (he/him)]

From a developer's perspective, it is preferable to know whether the problem still exists with as recent a version as possible.  So, yes, reproducing a bug with nightlies is ideal.  Knowing a bug happens on a recent thunderbird 3 beta is useful, but we would still want it duplicated with a nightly.  Duplicating bugs and getting info on problems with Thunderbird 2 is unlikely to be of any use because so much has changed since Thunderbird 2 (and non-security issues are unlikely to be addressed on 2.)

In terms of the specific bugs you cite, the default threading behavior has changed in Thunderbird 3.  Although bug 452221 is not resolved, users are unlikely to notice because we should no longer attempt to thread messages by their subject when there is no "Re:" present.  So yes, I would give a nightly a try in those cases, although that change probably happened in time for beta 2 (I'm not sure, though).

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #7)
Roger. My "same subject" in Comment #6 contains "Re: in subject" case. Andrew, I'll try to test issues in Comment #6 again using latest-trunk, with threading related setting variations (such as mail.strict_threading, settings relates to "Re: in subject header").

Comment 9

[Bob Noseworthy]

update - tried 2.0.0.21 on win xp, default install of thunderbird,  no plugins, same problem.

updated xp pro machine to thunderbird 3 beta 2 (using inbox from 2.0.0.21 install), same problem.  with a sortby->threaded, status bar indicates 'searching...' and fails to return on some searchs (hangs, indicates 'not responding', waited ~5min)   

NOTE, in all cases the search returns the results, possibly the complete set of search results and then hangs.

-----------
Same problem with todays nightly  (3.0b3pre)    
still works fine if NOT threaded,  still hangs.  Hangs with safe-mode version as well.  
-----------
Is there a quick way to sterilize my inbox and provide it to you folks for further investigation?   I suspect it has to do with the structure of the threading and the given search term, possibly combined with the volume of messages - or is there an easy way to capture further debug state without getting into a compiler/source code environment  (I can only help so much)

Comment 10

[Andrew Sutherland [:asuth] (he/him)]

Bob, thank you so much for trying out with the latest nightly!

Did you delete the .msf file for the folder or use the UI to force a rebuild of the folder with Thunderbird 3?  If not, could you give that a spin?  The .msf file stores persisted thread information, and this will help determine if the Thunderbird 3 threading is fine, but tricked by the persisted information, or whether it's something Thunderbird 3 is entirely broken on.

We could probably anonymize the msf sufficiently by passing it through a simple perl or python script or something...

Comment 11

[Bob Noseworthy]

rebuilt msf,  didn't fix it.

(shutdown thunderbird,  found the right profile, deleted inbox.msf,  restarted shredder, connected to my IMAP server and rebuilt the msf -- reperformed a search that causes the problem (in threaded view), and once again see shredder hang)


Would you only need the msf?  if the msf subjects can be scrubbed and the sender sufficiently munged (without randomizing it fully, so a similar problem-causing search can be launched)  then I could run such a script, check it, and submit it as an attachment for confirmation

Comment 12

[David :Bienvenu]

Bob, you can send me the .msf file, along with the string you're trying to do the quick search on, and I can see if it hangs here. I won't attach it to the bug...

Comment 13

[David :Bienvenu]

patch upcoming, thx for the msf file, Bob.

Comment 14

[David :Bienvenu]

Attachment: proposed fix

this makes us break out of the loop when we think the nesting is deeper than the number of messages in the thread. We may have a mis-displayed thread in the problem case, but it's better than hanging.

Comment 15

[David :Bienvenu]

fix checked in.

Comment 16

[Bob Noseworthy]

Hi David,  Bob here again -- just moved up to Thunderbird 3 RC1,  and this bug is back.  
When I have the quick search set to "Subject, From, or Recipient filter", and my large Inbox set to threaded view,  certain searches (specifically "ja") crashes Thunderbird 3,  just as it has with all previous thunderbirds except for the version fixed immediately after this bug was resolved.  I suspect the fix was lost, or a similar problem has crept in. 

Three crash reports were just submitted (logged under [e-mail], rather than this gmail address) 

This is 100% reproducible, application crash 100% of the time. 

Please let me know how I can help resolve this (my gmail addy is better for faster response on this)

Comment 17

[Wayne Mery (:wsmwk)]

(In reply to comment #16)
> Three crash reports were just submitted (logged under [e-mail], rather
> than this gmail address) 

you need to cite report ids, because we can't search by email address (which you don't what to post in a bug anyway).

perhaps one is bp-aae6b92b-9437-471d-8208-7d6e62091129
0	mozcrt19.dll	arena_malloc_small	 objdir-tb/mozilla/memory/jemalloc/src/jemalloc.c:4025
1	mozcrt19.dll	malloc	objdir-tb/mozilla/memory/jemalloc/src/jemalloc.c:6177
2	mozcrt19.dll	operator new	objdir-tb/mozilla/memory/jemalloc/src/new.cpp:54
3	thunderbird.exe	orkinHeap::Alloc	db/mork/src/orkinHeap.cpp:90
4	thunderbird.exe	morkNode::MakeNew	db/mork/src/morkNode.cpp:182
5	thunderbird.exe	morkTable::NewTableRowCursor	db/mork/src/morkTable.cpp:1540
6	thunderbird.exe	morkTable::GetTableRowCursor	db/mork/src/morkTable.cpp:458 
7	thunderbird.exe	nsMsgThread::GetChildHdrAt	 mailnews/db/msgdb/src/nsMsgThread.cpp:533
8	thunderbird.exe	nsMsgThread::GetChildHdrForKey	mailnews/db/msgdb/src/nsMsgThread.cpp:1069
9	thunderbird.exe	nsMsgThread::GetRootHdr	mailnews/db/msgdb/src/nsMsgThread.cpp:956
10	thunderbird.exe	nsMsgThreadEnumerator::nsMsgThreadEnumerator	mailnews/db/msgdb/src/nsMsgThread.cpp:699
11	thunderbird.exe	nsMsgThread::EnumerateMessages	mailnews/db/msgdb/src/nsMsgThread.cpp:905
12	thunderbird.exe	nsMsgQuickSearchDBView::ListIdsInThreadOrder	mailnews/base/src/nsMsgQuickSearchDBView.cpp:643
13	thunderbird.exe	nsMsgQuickSearchDBView::ListIdsInThreadOrder	mailnews/base/src/nsMsgQuickSearchDBView.cpp:669
14	thunderbird.exe	nsMsgQuickSearchDBView::ListIdsInThreadOrder	mailnews/base/src/nsMsgQuickSearchDBView.cpp:669 
...
16159	thunderbird.exe	nsMsgQuickSearchDBView::ListIdsInThreadOrder	 mailnews/base/src/nsMsgQuickSearchDBView.cpp:669
16160	thunderbird.exe	nsMsgQuickSearchDBView::ListIdsInThreadOrder	mailnews/base/src/nsMsgQuickSearchDBView.cpp:682
16161	thunderbird.exe	nsMsgQuickSearchDBView::SortThreads	mailnews/base/src/nsMsgQuickSearchDBView.cpp:533
16162	thunderbird.exe	nsMsgThreadedDBView::Sort	mailnews/base/src/nsMsgThreadedDBView.cpp:361
16163	thunderbird.exe	nsMsgQuickSearchDBView::OnSearchDone	mailnews/base/src/nsMsgQuickSearchDBView.cpp:337
16164	thunderbird.exe	nsMsgSearchSession::NotifyListenersDone	mailnews/base/search/src/nsMsgSearchSession.cpp:598
16165	thunderbird.exe	nsMsgSearchSession::TimerCallback	mailnews/base/search/src/nsMsgSearchSession.cpp:538
16166	xpcom_core.dll	nsTimerImpl::Fire	xpcom/threads/nsTimerImpl.cpp:420 

Perhaps your crash may be a regression of this bug (I have doubts), but in any event it is not "this bug".  So re-resolving FIXED.  When new/changed symptoms appear, the practice is to file a new bug.  Please do so. (And if the bug is a regression, we set it to block the bug which caused it.)

but if the above report id is yours, then your crash is bug 530044