347665 - Opening a very large IMAP folder with a large number of new messages takes a long time

Status: RESOLVED WORKSFORME

(MailNews Core :: Database, defect)

Description

[Samuel Sieb]

I have an imap folder with over 100,000 messages in it.  It's in threaded mode.  I don't know how to switch it to non-threaded mode without opening it.  On an Athlon 64, I let it run for two hours and it didn't come anywhere near finishing.  I eventually intervened with a debugger.  I have stack traces which I will post soon.

Comment 1

[Samuel Sieb]

Here's where the problem is:
#0  nsMsgKeyArray::FindIndex (this=0xac977cc, key=100264, startIndex=1126)
    at nsMsgKeyArray.cpp:49
#1  0x017e2c1a in nsMsgDBView::FindParentInThread (this=0xac977a0,
    parentKey=100264, startOfThreadViewIndex=1126) at nsMsgDBView.cpp:4293
#2  0x017f1b6f in nsMsgThreadedDBView::OnNewHeader (this=0xac977a0,
    newHdr=0xf4738c0, aParentKey=100264, ensureListed=0)
    at nsMsgThreadedDBView.cpp:627
#3  0x017dd34a in nsMsgDBView::OnHdrAdded (this=0xac977a0,
    aHdrChanged=0xf4738c0, aParentKey=100264, aFlags=65536, aInstigator=0x0)
    at nsMsgDBView.cpp:4532
#4  0x00c46682 in nsMsgDatabase::NotifyHdrAddedAll (this=0xaa59bc8,
    aHdrAdded=0xf4738c0, parentKey=100264, flags=65536, instigator=0x0)
    at nsMsgDatabase.cpp:686
#5  0x00c43bb7 in nsMsgDatabase::AddNewHdrToDB (this=0xaa59bc8,
    newHdr=0xf4738c0, notify=1) at nsMsgDatabase.cpp:3053
#6  0x00c5da80 in nsImapMailDatabase::AddNewHdrToDB (this=0xaa59bc8,
    newHdr=0xf4738c0, notify=1) at nsImapMailDatabase.cpp:144
#7  0x03179272 in nsImapMailFolder::NormalEndHeaderParseStream (
    this=0xa41f2f8, aProtocol=0xe3a4858, imapUrl=0xe398ba8)
    at nsImapMailFolder.cpp:3065
#8  0x0317a133 in nsImapMailFolder::ParseMsgHdrs (this=0xa41f2f8,
    aProtocol=0xe3a4858, aHdrXferInfo=0xe3a4b40) at nsImapMailFolder.cpp:2883

Line numbers of course will be a little off due to changes in the various files, but the function names are there.  The problem is that nsMsgKeyArray::FindIndex does a linear search, so the end result is that the threading takes forever.  The list appears to be created in the same order as it will be shown in the display, so would it be possible to have a second array that is always in index order, so that it could be binary searched?

Comment 2

[David :Bienvenu]

are there a lot of messages with the same subject? If so, this is fixed in 2.0, so that we don't try to thread within a thread by subject if there are a large number of messages with the same subject.

Comment 3

[Samuel Sieb]

2.0 of what?

Comment 4

[David :Bienvenu]

2.0 of Thunderbird, 1.5 of Seamonkey ( or maybe 1.1 of Seamonkey - I'm not sure of their version numbers).

Comment 5

[Justin Wood (:Callek)]

unless things change (unlikely), that will be SeaMonkey 1.1 to match backend of Thunderbird 2.0

Comment 6

[Samuel Sieb]

Attachment: use sortedIndex instead of Index lookup

This is patch for a smaller problem that I ran into.  The function that adds to the m_new array says that the array is always in sorted order and I haven't had any problems with this patch in the month that I've used it.

Comment 7

[David :Bienvenu]

Comment on attachment 233199 [details] [diff] [review]
use sortedIndex instead of Index lookup

looks good; I'll try it out and check it in. thx!

Comment 8

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #7)
> (From update of attachment 233199 [details] [diff] [review] [edit])
> looks good; I'll try it out and check it in. thx!

Doesn't look like this landed on the trunk, David.

Comment 9

[Wayne Mery (:wsmwk)]

David in comment #7
> (From update of attachment 233199 [details] [diff] [review] [edit])
> looks good; I'll try it out and check it in. thx!

David this still needs checkin AFAICT

 
David in comment #2:
> are there a lot of messages with the same subject? If so, fixed in 2.0

David, 
* is the 2.0 fix a change to mail.thread_without_re per bug 90452, or a different bug?
* will the 2.0 change and perhaps this bug also help mailnews with big threads, possibilities are bug 202246, bug 77872, bug 84042

Comment 10

[David :Bienvenu]

landed this on the trunk. If it shakes out OK, I'll land this for 2.0 after we unfreeze after beta 2.

Comment 11

[David :Bienvenu]

landed on 2.0 beta branch

Comment 12

[Jo Hermans]

Waw! Version 2 beta 2 (20070130) is now a /lot/ faster when opening large folders!

Comment 13

[David :Bienvenu]

cool - I assume you mean large imap folders with lots of new messages? This fix only speeds up dealing with a large number of new messages.

Comment 14

[Samuel Sieb]

This isn't completely fixed, that was just a small fix.  There is still a bigger problem with large folders (at least mine).

Comment 15

[David :Bienvenu]

if you're talking about the time it takes the server to respond to the request for all the message flags in a folder, that's a separate issue, and is covered by other bugs.

Comment 16

[Wayne Mery (:wsmwk)]

mod summary

Comment 17

[Jo Hermans]

(In reply to comment #13)
> cool - I assume you mean large imap folders with lots of new messages? This fix
> only speeds up dealing with a large number of new messages.
> 

Yes, I have a few big ones (1 over 6000 messages, and a few over 1000 messages), into which most of my new messages are moved by filters (my inbox is almost empty, and only hold the ones that weren't caught by a filter). Opening such a large box when there are many new messages, is much faster now.

Comment 18

[Samuel Sieb]

I'm referring to comment 0 and 1.  The patch I supplied was a small speedup I found in the process of trying to figure out the main problem.

Comment 19

[Wayne Mery (:wsmwk)]

(In reply to comment #18)
> I'm referring to comment 0 and 1.  The patch I supplied was a small speedup I
> found in the process of trying to figure out the main problem.

So would you say my change in summary is not accurate? 
We don't yet know the cause of the main problem of "large folder" and it isn't addressed in other bugs? 

Comment 20

[Samuel Sieb]

The summary change is fine.  The issue is not resolved though.

Comment 21

[Samuel Sieb]

Oh, and as far as I can tell, it's not addressed in any other bug.

Comment 22

[Wayne Mery (:wsmwk)]

Samuel, is the remaining issue "The list appears to be created in the same order as it will be shown in the display, so would it be possible to have a second array that is always in index order, so that it could be binary searched?"

Comment 23

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> I have an imap folder with over 100,000 messages in it. It's in threaded mode.

Very long and/or very deep thread case?
  - very many mails of same subject including subject with 'Re:'
  - very many mails in a thread by In-Reply-To:
There is at least following issues when very long/deep thread.
 (A) Rebuild Index takes long.
 (B) Change from "non thread view" to "thread view" takes long.
     Bug 226730
 (C) Mail deletion takes long.
     Bug 429753, which was closed as 296453.

To Samuel Sieb(bug opener):
If all mails of your "large number of new messages" has same subject,
issue of (a) occurs because new mails are added(opposite problem to Bug 429753), 
and problem of (b) occurs due to view refresh because "View/Thread" is already chosen in your case.
Does "Very long and/or very deep thread" is involved in your case?

Comment 24

[WADA:World Anti-bad-Duping Agency]

I wanted to say(sorry for spam):
  "Very long and/or very deep thread" is involved in your case?

Comment 25

[Samuel Sieb]

Sorry, I haven't been reading my bugmail.  My test case is the kernel mailing list.  It has a few deep threads, but not a lot of the same subject otherwise.  I don't have enough emails any more to test the conditions that I had before, but I haven't had any problems recently.  So maybe it's been fixed somehow or people just don't have such big mail folders.  I'll try to collect enough to test this out, but it will take a while.

Comment 26

[Ludovic Hirlimann [:Usul]]

(In reply to comment #25)
> Sorry, I haven't been reading my bugmail.  My test case is the kernel mailing
> list.  It has a few deep threads, but not a lot of the same subject otherwise. 
> I don't have enough emails any more to test the conditions that I had before,
> but I haven't had any problems recently.  So maybe it's been fixed somehow or
> people just don't have such big mail folders.  I'll try to collect enough to
> test this out, but it will take a while.

Hi Sam - when you update the bug can you state which version of TB you used while doinG your tests ?

Comment 27

[Samuel Sieb]

I use Seamonkey, not Thunderbird.  I just did another very large test and there was no problem at all with Seamonkey 2.0.

Comment 28

[Ludovic Hirlimann [:Usul]]

Closing WFM per comment #27