Last Comment Bug 347665 - Opening a very large IMAP folder with a large number of new messages takes a long time
: Opening a very large IMAP folder with a large number of new messages takes a ...
Status: RESOLVED WORKSFORME
: fixed1.8.1.2, helpwanted, perf
Product: MailNews Core
Classification: Components
Component: Database (show other bugs)
: Trunk
: x86 All
: -- normal with 2 votes (vote)
: ---
Assigned To: Samuel Sieb
:
Mentors:
Depends on:
Blocks: 384360
  Show dependency treegraph
 
Reported: 2006-08-06 16:43 PDT by Samuel Sieb
Modified: 2010-02-04 05:41 PST (History)
13 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
use sortedIndex instead of Index lookup (791 bytes, patch)
2006-08-10 20:15 PDT, Samuel Sieb
mozilla: superreview+
Details | Diff | Splinter Review

Description Samuel Sieb 2006-08-06 16:43:09 PDT
I have an imap folder with over 100,000 messages in it.  It's in threaded mode.  I don't know how to switch it to non-threaded mode without opening it.  On an Athlon 64, I let it run for two hours and it didn't come anywhere near finishing.  I eventually intervened with a debugger.  I have stack traces which I will post soon.
Comment 1 Samuel Sieb 2006-08-06 20:43:27 PDT
Here's where the problem is:
#0  nsMsgKeyArray::FindIndex (this=0xac977cc, key=100264, startIndex=1126)
    at nsMsgKeyArray.cpp:49
#1  0x017e2c1a in nsMsgDBView::FindParentInThread (this=0xac977a0,
    parentKey=100264, startOfThreadViewIndex=1126) at nsMsgDBView.cpp:4293
#2  0x017f1b6f in nsMsgThreadedDBView::OnNewHeader (this=0xac977a0,
    newHdr=0xf4738c0, aParentKey=100264, ensureListed=0)
    at nsMsgThreadedDBView.cpp:627
#3  0x017dd34a in nsMsgDBView::OnHdrAdded (this=0xac977a0,
    aHdrChanged=0xf4738c0, aParentKey=100264, aFlags=65536, aInstigator=0x0)
    at nsMsgDBView.cpp:4532
#4  0x00c46682 in nsMsgDatabase::NotifyHdrAddedAll (this=0xaa59bc8,
    aHdrAdded=0xf4738c0, parentKey=100264, flags=65536, instigator=0x0)
    at nsMsgDatabase.cpp:686
#5  0x00c43bb7 in nsMsgDatabase::AddNewHdrToDB (this=0xaa59bc8,
    newHdr=0xf4738c0, notify=1) at nsMsgDatabase.cpp:3053
#6  0x00c5da80 in nsImapMailDatabase::AddNewHdrToDB (this=0xaa59bc8,
    newHdr=0xf4738c0, notify=1) at nsImapMailDatabase.cpp:144
#7  0x03179272 in nsImapMailFolder::NormalEndHeaderParseStream (
    this=0xa41f2f8, aProtocol=0xe3a4858, imapUrl=0xe398ba8)
    at nsImapMailFolder.cpp:3065
#8  0x0317a133 in nsImapMailFolder::ParseMsgHdrs (this=0xa41f2f8,
    aProtocol=0xe3a4858, aHdrXferInfo=0xe3a4b40) at nsImapMailFolder.cpp:2883

Line numbers of course will be a little off due to changes in the various files, but the function names are there.  The problem is that nsMsgKeyArray::FindIndex does a linear search, so the end result is that the threading takes forever.  The list appears to be created in the same order as it will be shown in the display, so would it be possible to have a second array that is always in index order, so that it could be binary searched?
Comment 2 David :Bienvenu 2006-08-07 08:52:14 PDT
are there a lot of messages with the same subject? If so, this is fixed in 2.0, so that we don't try to thread within a thread by subject if there are a large number of messages with the same subject.
Comment 3 Samuel Sieb 2006-08-07 21:15:14 PDT
2.0 of what?
Comment 4 David :Bienvenu 2006-08-07 21:57:39 PDT
2.0 of Thunderbird, 1.5 of Seamonkey ( or maybe 1.1 of Seamonkey - I'm not sure of their version numbers).
Comment 5 Justin Wood (:Callek) (Away until Aug 29) 2006-08-08 06:52:10 PDT
unless things change (unlikely), that will be SeaMonkey 1.1 to match backend of Thunderbird 2.0
Comment 6 Samuel Sieb 2006-08-10 20:15:23 PDT
Created attachment 233199 [details] [diff] [review]
use sortedIndex instead of Index lookup

This is patch for a smaller problem that I ran into.  The function that adds to the m_new array says that the array is always in sorted order and I haven't had any problems with this patch in the month that I've used it.
Comment 7 David :Bienvenu 2006-08-10 20:25:55 PDT
Comment on attachment 233199 [details] [diff] [review]
use sortedIndex instead of Index lookup

looks good; I'll try it out and check it in. thx!
Comment 8 Stephen Donner [:stephend] 2006-09-23 20:58:07 PDT
(In reply to comment #7)
> (From update of attachment 233199 [details] [diff] [review] [edit])
> looks good; I'll try it out and check it in. thx!

Doesn't look like this landed on the trunk, David.
Comment 9 Wayne Mery (:wsmwk, NI for questions) 2006-11-20 11:38:10 PST
David in comment #7
> (From update of attachment 233199 [details] [diff] [review] [edit])
> looks good; I'll try it out and check it in. thx!

David this still needs checkin AFAICT

 
David in comment #2:
> are there a lot of messages with the same subject? If so, fixed in 2.0

David, 
* is the 2.0 fix a change to mail.thread_without_re per bug 90452, or a different bug?
* will the 2.0 change and perhaps this bug also help mailnews with big threads, possibilities are bug 202246, bug 77872, bug 84042
Comment 10 David :Bienvenu 2007-01-16 13:49:08 PST
landed this on the trunk. If it shakes out OK, I'll land this for 2.0 after we unfreeze after beta 2.
Comment 11 David :Bienvenu 2007-01-29 09:11:50 PST
landed on 2.0 beta branch
Comment 12 Jo Hermans 2007-01-30 09:29:05 PST
Waw! Version 2 beta 2 (20070130) is now a /lot/ faster when opening large folders!
Comment 13 David :Bienvenu 2007-01-30 09:42:15 PST
cool - I assume you mean large imap folders with lots of new messages? This fix only speeds up dealing with a large number of new messages.
Comment 14 Samuel Sieb 2007-01-30 09:45:00 PST
This isn't completely fixed, that was just a small fix.  There is still a bigger problem with large folders (at least mine).
Comment 15 David :Bienvenu 2007-01-30 09:50:48 PST
if you're talking about the time it takes the server to respond to the request for all the message flags in a folder, that's a separate issue, and is covered by other bugs.
Comment 16 Wayne Mery (:wsmwk, NI for questions) 2007-01-30 09:59:37 PST
mod summary
Comment 17 Jo Hermans 2007-01-30 10:59:17 PST
(In reply to comment #13)
> cool - I assume you mean large imap folders with lots of new messages? This fix
> only speeds up dealing with a large number of new messages.
> 

Yes, I have a few big ones (1 over 6000 messages, and a few over 1000 messages), into which most of my new messages are moved by filters (my inbox is almost empty, and only hold the ones that weren't caught by a filter). Opening such a large box when there are many new messages, is much faster now.
Comment 18 Samuel Sieb 2007-01-30 16:06:07 PST
I'm referring to comment 0 and 1.  The patch I supplied was a small speedup I found in the process of trying to figure out the main problem.
Comment 19 Wayne Mery (:wsmwk, NI for questions) 2007-08-25 08:27:19 PDT
(In reply to comment #18)
> I'm referring to comment 0 and 1.  The patch I supplied was a small speedup I
> found in the process of trying to figure out the main problem.

So would you say my change in summary is not accurate? 
We don't yet know the cause of the main problem of "large folder" and it isn't addressed in other bugs? 
Comment 20 Samuel Sieb 2007-08-27 13:04:09 PDT
The summary change is fine.  The issue is not resolved though.
Comment 21 Samuel Sieb 2007-08-27 13:05:07 PDT
Oh, and as far as I can tell, it's not addressed in any other bug.
Comment 22 Wayne Mery (:wsmwk, NI for questions) 2008-05-28 11:48:19 PDT
Samuel, is the remaining issue "The list appears to be created in the same order as it will be shown in the display, so would it be possible to have a second array that is always in index order, so that it could be binary searched?"
Comment 23 WADA 2008-05-28 14:20:47 PDT
(In reply to comment #0)
> I have an imap folder with over 100,000 messages in it. It's in threaded mode.

Very long and/or very deep thread case?
  - very many mails of same subject including subject with 'Re:'
  - very many mails in a thread by In-Reply-To:
There is at least following issues when very long/deep thread.
 (A) Rebuild Index takes long.
 (B) Change from "non thread view" to "thread view" takes long.
     Bug 226730
 (C) Mail deletion takes long.
     Bug 429753, which was closed as 296453.

To Samuel Sieb(bug opener):
If all mails of your "large number of new messages" has same subject,
issue of (a) occurs because new mails are added(opposite problem to Bug 429753), 
and problem of (b) occurs due to view refresh because "View/Thread" is already chosen in your case.
Does "Very long and/or very deep thread" is involved in your case?
Comment 24 WADA 2008-05-28 15:14:26 PDT
I wanted to say(sorry for spam):
  "Very long and/or very deep thread" is involved in your case?
Comment 25 Samuel Sieb 2008-10-12 21:12:47 PDT
Sorry, I haven't been reading my bugmail.  My test case is the kernel mailing list.  It has a few deep threads, but not a lot of the same subject otherwise.  I don't have enough emails any more to test the conditions that I had before, but I haven't had any problems recently.  So maybe it's been fixed somehow or people just don't have such big mail folders.  I'll try to collect enough to test this out, but it will take a while.
Comment 26 Ludovic Hirlimann [:Usul] 2009-04-15 04:18:49 PDT
(In reply to comment #25)
> Sorry, I haven't been reading my bugmail.  My test case is the kernel mailing
> list.  It has a few deep threads, but not a lot of the same subject otherwise. 
> I don't have enough emails any more to test the conditions that I had before,
> but I haven't had any problems recently.  So maybe it's been fixed somehow or
> people just don't have such big mail folders.  I'll try to collect enough to
> test this out, but it will take a while.

Hi Sam - when you update the bug can you state which version of TB you used while doinG your tests ?
Comment 27 Samuel Sieb 2010-01-27 15:04:35 PST
I use Seamonkey, not Thunderbird.  I just did another very large test and there was no problem at all with Seamonkey 2.0.
Comment 28 Ludovic Hirlimann [:Usul] 2010-02-04 05:41:16 PST
Closing WFM per comment #27

Note You need to log in before you can comment on or make changes to this bug.