Opening a very large IMAP folder with a large number of new messages takes a long time

RESOLVED WORKSFORME

Status

MailNews Core
Database
RESOLVED WORKSFORME
11 years ago
8 years ago

People

(Reporter: Samuel Sieb, Assigned: Samuel Sieb)

Tracking

(Blocks: 1 bug, {fixed1.8.1.2, helpwanted, perf})

Trunk
x86
All
fixed1.8.1.2, helpwanted, perf

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

11 years ago
I have an imap folder with over 100,000 messages in it.  It's in threaded mode.  I don't know how to switch it to non-threaded mode without opening it.  On an Athlon 64, I let it run for two hours and it didn't come anywhere near finishing.  I eventually intervened with a debugger.  I have stack traces which I will post soon.
(Assignee)

Comment 1

11 years ago
Here's where the problem is:
#0  nsMsgKeyArray::FindIndex (this=0xac977cc, key=100264, startIndex=1126)
    at nsMsgKeyArray.cpp:49
#1  0x017e2c1a in nsMsgDBView::FindParentInThread (this=0xac977a0,
    parentKey=100264, startOfThreadViewIndex=1126) at nsMsgDBView.cpp:4293
#2  0x017f1b6f in nsMsgThreadedDBView::OnNewHeader (this=0xac977a0,
    newHdr=0xf4738c0, aParentKey=100264, ensureListed=0)
    at nsMsgThreadedDBView.cpp:627
#3  0x017dd34a in nsMsgDBView::OnHdrAdded (this=0xac977a0,
    aHdrChanged=0xf4738c0, aParentKey=100264, aFlags=65536, aInstigator=0x0)
    at nsMsgDBView.cpp:4532
#4  0x00c46682 in nsMsgDatabase::NotifyHdrAddedAll (this=0xaa59bc8,
    aHdrAdded=0xf4738c0, parentKey=100264, flags=65536, instigator=0x0)
    at nsMsgDatabase.cpp:686
#5  0x00c43bb7 in nsMsgDatabase::AddNewHdrToDB (this=0xaa59bc8,
    newHdr=0xf4738c0, notify=1) at nsMsgDatabase.cpp:3053
#6  0x00c5da80 in nsImapMailDatabase::AddNewHdrToDB (this=0xaa59bc8,
    newHdr=0xf4738c0, notify=1) at nsImapMailDatabase.cpp:144
#7  0x03179272 in nsImapMailFolder::NormalEndHeaderParseStream (
    this=0xa41f2f8, aProtocol=0xe3a4858, imapUrl=0xe398ba8)
    at nsImapMailFolder.cpp:3065
#8  0x0317a133 in nsImapMailFolder::ParseMsgHdrs (this=0xa41f2f8,
    aProtocol=0xe3a4858, aHdrXferInfo=0xe3a4b40) at nsImapMailFolder.cpp:2883

Line numbers of course will be a little off due to changes in the various files, but the function names are there.  The problem is that nsMsgKeyArray::FindIndex does a linear search, so the end result is that the threading takes forever.  The list appears to be created in the same order as it will be shown in the display, so would it be possible to have a second array that is always in index order, so that it could be binary searched?

Comment 2

11 years ago
are there a lot of messages with the same subject? If so, this is fixed in 2.0, so that we don't try to thread within a thread by subject if there are a large number of messages with the same subject.
(Assignee)

Comment 3

11 years ago
2.0 of what?

Comment 4

11 years ago
2.0 of Thunderbird, 1.5 of Seamonkey ( or maybe 1.1 of Seamonkey - I'm not sure of their version numbers).

Comment 5

11 years ago
unless things change (unlikely), that will be SeaMonkey 1.1 to match backend of Thunderbird 2.0
(Assignee)

Comment 6

11 years ago
Created attachment 233199 [details] [diff] [review]
use sortedIndex instead of Index lookup

This is patch for a smaller problem that I ran into.  The function that adds to the m_new array says that the array is always in sorted order and I haven't had any problems with this patch in the month that I've used it.

Comment 7

11 years ago
Comment on attachment 233199 [details] [diff] [review]
use sortedIndex instead of Index lookup

looks good; I'll try it out and check it in. thx!
Attachment #233199 - Flags: superreview+

Updated

11 years ago
Assignee: bienvenu → samuel
Keywords: perf
Summary: opening a very large imap folder takes a long time → Opening a very large IMAP folder takes a long time
Whiteboard: [checkin needed]
(In reply to comment #7)
> (From update of attachment 233199 [details] [diff] [review] [edit])
> looks good; I'll try it out and check it in. thx!

Doesn't look like this landed on the trunk, David.
David in comment #7
> (From update of attachment 233199 [details] [diff] [review] [edit])
> looks good; I'll try it out and check it in. thx!

David this still needs checkin AFAICT

 
David in comment #2:
> are there a lot of messages with the same subject? If so, fixed in 2.0

David, 
* is the 2.0 fix a change to mail.thread_without_re per bug 90452, or a different bug?
* will the 2.0 change and perhaps this bug also help mailnews with big threads, possibilities are bug 202246, bug 77872, bug 84042

Comment 10

11 years ago
landed this on the trunk. If it shakes out OK, I'll land this for 2.0 after we unfreeze after beta 2.

Updated

11 years ago
Whiteboard: [checkin needed]

Comment 11

11 years ago
landed on 2.0 beta branch
Status: NEW → RESOLVED
Last Resolved: 11 years ago
Keywords: fixed1.8.1.2
Resolution: --- → FIXED

Comment 12

11 years ago
Waw! Version 2 beta 2 (20070130) is now a /lot/ faster when opening large folders!

Comment 13

11 years ago
cool - I assume you mean large imap folders with lots of new messages? This fix only speeds up dealing with a large number of new messages.
(Assignee)

Comment 14

11 years ago
This isn't completely fixed, that was just a small fix.  There is still a bigger problem with large folders (at least mine).
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Comment 15

11 years ago
if you're talking about the time it takes the server to respond to the request for all the message flags in a folder, that's a separate issue, and is covered by other bugs.
mod summary
Summary: Opening a very large IMAP folder takes a long time → Opening a very large IMAP folder with a large number of new messages takes a long time

Comment 17

11 years ago
(In reply to comment #13)
> cool - I assume you mean large imap folders with lots of new messages? This fix
> only speeds up dealing with a large number of new messages.
> 

Yes, I have a few big ones (1 over 6000 messages, and a few over 1000 messages), into which most of my new messages are moved by filters (my inbox is almost empty, and only hold the ones that weren't caught by a filter). Opening such a large box when there are many new messages, is much faster now.
(Assignee)

Comment 18

11 years ago
I'm referring to comment 0 and 1.  The patch I supplied was a small speedup I found in the process of trying to figure out the main problem.
Blocks: 384360
(In reply to comment #18)
> I'm referring to comment 0 and 1.  The patch I supplied was a small speedup I
> found in the process of trying to figure out the main problem.

So would you say my change in summary is not accurate? 
We don't yet know the cause of the main problem of "large folder" and it isn't addressed in other bugs? 
(Assignee)

Comment 20

10 years ago
The summary change is fine.  The issue is not resolved though.
(Assignee)

Comment 21

10 years ago
Oh, and as far as I can tell, it's not addressed in any other bug.
Samuel, is the remaining issue "The list appears to be created in the same order as it will be shown in the display, so would it be possible to have a second array that is always in index order, so that it could be binary searched?"
Keywords: helpwanted
(In reply to comment #0)
> I have an imap folder with over 100,000 messages in it. It's in threaded mode.

Very long and/or very deep thread case?
  - very many mails of same subject including subject with 'Re:'
  - very many mails in a thread by In-Reply-To:
There is at least following issues when very long/deep thread.
 (A) Rebuild Index takes long.
 (B) Change from "non thread view" to "thread view" takes long.
     Bug 226730
 (C) Mail deletion takes long.
     Bug 429753, which was closed as 296453.

To Samuel Sieb(bug opener):
If all mails of your "large number of new messages" has same subject,
issue of (a) occurs because new mails are added(opposite problem to Bug 429753), 
and problem of (b) occurs due to view refresh because "View/Thread" is already chosen in your case.
Does "Very long and/or very deep thread" is involved in your case?
I wanted to say(sorry for spam):
  "Very long and/or very deep thread" is involved in your case?
Product: Core → MailNews Core
(Assignee)

Comment 25

9 years ago
Sorry, I haven't been reading my bugmail.  My test case is the kernel mailing list.  It has a few deep threads, but not a lot of the same subject otherwise.  I don't have enough emails any more to test the conditions that I had before, but I haven't had any problems recently.  So maybe it's been fixed somehow or people just don't have such big mail folders.  I'll try to collect enough to test this out, but it will take a while.
(In reply to comment #25)
> Sorry, I haven't been reading my bugmail.  My test case is the kernel mailing
> list.  It has a few deep threads, but not a lot of the same subject otherwise. 
> I don't have enough emails any more to test the conditions that I had before,
> but I haven't had any problems recently.  So maybe it's been fixed somehow or
> people just don't have such big mail folders.  I'll try to collect enough to
> test this out, but it will take a while.

Hi Sam - when you update the bug can you state which version of TB you used while doinG your tests ?
(Assignee)

Comment 27

8 years ago
I use Seamonkey, not Thunderbird.  I just did another very large test and there was no problem at all with Seamonkey 2.0.
Closing WFM per comment #27
Status: REOPENED → RESOLVED
Last Resolved: 11 years ago8 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.