Closed Bug 496318 Opened 15 years ago Closed 15 years ago

Archiving large amounts of mail causes crash

Categories

(Thunderbird :: Mail Window Front End, defect)

x86
Windows XP
defect
Not set
critical

Tracking

(Not tracked)

VERIFIED FIXED
Thunderbird 3.0b3

People

(Reporter: brian, Assigned: Bienvenu)

References

Details

(Keywords: crash, perf, regression)

Attachments

(4 files)

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1pre) Gecko/20090603 Lightning/1.0pre Shredder/3.0b3pre

Selecting 200+ mails for archiving in one go causes TB to hang/crash. Smaller batches just seem very slow. Unless performance is improved here I can see it taking some time to archive large mailboxes.

Sorry if I post this incorrectly it is my first bugzilla post :)

Reproducible: Always

Steps to Reproduce:
1.Selcet 200+ mails in some mail folder (eg INBOX)
2.Click archive button (or A)
3.
Actual Results:  
TB either becomes unresponsive or generates M$ crash dialogue box

Expected Results:  
Should just archive mail
 
> TB either becomes unresponsive or generates M$ crash dialogue box

so the M$ crash dialog comes from you trying to kill the unresponsive application ?
 
> Expected Results:  
> Should just archive mail

Is breakpad enabled ? if not can you make sure to enable it and give us a crashid (see http://kb.mozillazine.org/Breakpad) for more information.
Severity: normal → critical
Keywords: crash, perf
Version: unspecified → Trunk
I have tested this more rigorously and the problem seems to be related to memory use, admittedly with rather more mails than I initially indicated.

I tried archiving 2300 mails whilst watching RAM/pagefile usage, I repeated this twice, both times TB crashed without generating any errors (just disappeared from screen and task manager). The crash point appears to be when the memory usage of TB approaches the free RAM, in my case this was at 1800MB as the attached screenshots show. The two crashes coincide with the peaks of the two 'ramps' in the PF Usage graph, the second graph was grabbed at the point TB crashed.

The previous crashes may have been related to TB carrying out much reindexing on a newly migrated installation (TB2 - 3) with 10k+ mails in 20+ folders and are unreproducible at the present. Though I did submit a crash report yesterday (e39a1f76-b800-4e58-bbd3-0fc1e2090204) on the one occasion it popped up, the other three occasions I just got the M$ crash dialogue (event log entry below).

I hope all this helps you, if I can give you any more information just ask.
Whiteboard: did submit a crash report yesterday (e39a1f76-b800-4e58-bbd3-0fc1e2090204) on the one occasion it popped up, the other three occasions I just got the M$ crash dialogue (event log entry below). I hope all this helps you I repeated this twice, both times T…
Attached image Crash Event screenshot
Whiteboard: did submit a crash report yesterday (e39a1f76-b800-4e58-bbd3-0fc1e2090204) on the one occasion it popped up, the other three occasions I just got the M$ crash dialogue (event log entry below). I hope all this helps you in my case this was at 1800MB as th… → Though I did submit a crash report yesterday (e39a1f76-b800-4e58-bbd3-0fc1e2090204) on the one occasion it popped up, the other three occasions I just got the M$ crash dialogue (Crash Event screenshot). I hope all this helps you in my case this was at 18…
(In reply to comment #2)
> I have tested this more rigorously and the problem seems to be related to
> memory use, admittedly with rather more mails than I initially indicated.

Do you use pop3 or imap ?

> (e39a1f76-b800-4e58-bbd3-0fc1e2090204) on the one occasion it popped up, the
> other three occasions I just got the M$ crash dialogue (event log entry below).
> 
> I hope all this helps you, if I can give you any more information just ask.

does not decode at the moment  but yes it helps.
Whiteboard: I repeated this twice, both times TB crashed without generating any errors (just disappeared from screen and task manager). The crash point appears to be when the memory usage of TB approaches the free RAM if I can give you any more information just ask. …
Woo, this doesn't sound good. I know archiving is slow with 80 to 100 messages but probably the crash is due to OOM.
the archive code doesn't inherently use much memory. I would suspect that some other code besides the move/copy code (which is what archive uses) notices all the messages getting moved and kicks off some process that allocates a lot of memory, e.g., indexing or imap auto sync.
> Do you use pop3 or imap ?

 my stores are IMAP on courier IMAP server

> I would suspect that some other code besides the move/copy code

I have noticed some re index/resynchronise messages flying about during the archive process but as the system and activity manager are slow to respond under this situation it's hard to tell what's going on. I do know that it only takes a minute or two to max out the memory, quite impressive the rate that it gets used!
(In reply to comment #6)

> > (e39a1f76-b800-4e58-bbd3-0fc1e2090204) on the one occasion it popped up, the
> > other three occasions I just got the M$ crash dialogue (event log entry below).
> > 
> > I hope all this helps you, if I can give you any more information just ask.
> 
> does not decode at the moment  but yes it helps.

This is because of bug Bug 495275.
Keywords: qawanted
OK, we're summarizing the selection over and over and over again, for every message that gets deleted. We should be suppressing selection while the archive is doing batch deletes.
Assignee: nobody → bienvenu
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Flags: blocking-thunderbird3+
Target Milestone: --- → Thunderbird 3.0b3
> We should be suppressing selection while the archive is doing batch deletes.

And my guess is that that would give a significant performance increase when archiving. Two birds, one stone.
Keywords: qawanted
Blocks: 454829
Keywords: regression
Bug 495275 is about the dump not being decoded at the moment.
Attached patch proposed fixSplinter Review
as discussed on irc, this makes us avoid re-summarizing a selection if the number of messages selected has changed in the last 100 mSec. This fixes this bug; it will also help with the case of the user holding down the arrow key to extend the selection. Asking for davida review since he knows this code the best.
Attachment #381582 - Flags: review?(david.ascher)
I can confirm that w/o this patch, if you archive 2000 messages, we do consume gigabytes of memory.  I'm not sure if it's simply bloat of trying to stream the selected messages 2000 times, or if GC can't keep up, or there's an actual memory leak somewhere.
Attachment #381582 - Flags: review?(david.ascher) → review+
Comment on attachment 381582 [details] [diff] [review]
proposed fix

Just a few nits:

> 
>-  summarizeSelection: function()
>+ // This method either handles the selection, or sets a timer to handle
>+ // it once it stops changing.

Ideally that'd be a doxygen comment.

>+  showSummary: function(aThis, aSelCount)
>   {
>+    dump("in show summary, aSelCount = " + aSelCount + "\n");

Remove the dump() call.


>+    // If we are already summarized, let's make sure the selection count
>+    // isn't changing rapidly, by checking again in 100 mSec.

it's actually spelled "msec", afaik.

as long as the dump() is gone, r=davida.
fix checked in, with nits addressed. I think this might also incidentally fix the delete/archive messages and then land on a collapsed thread issue.
Status: ASSIGNED → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Well, gratifying to know I found a bug and you guys fixed it so promptly. Does this fix get rolled into a nightly now? If so where can I view the changelog for TB3?
No longer blocks: 454829
(In reply to comment #18)
> Well, gratifying to know I found a bug and you guys fixed it so promptly. Does
> this fix get rolled into a nightly now? If so where can I view the changelog
> for TB3?

It should be in the next nightly. 
Changelog can be seen at http://hg.mozilla.org/comm-central/ or on http://www.squarefree.com/burningedge/
I can confirm that this bug is now fixed. As I suspected archiving is now an order of magnitude faster (less than 2 mins for 1200 mails in my case, 45s for 500); also moving mails from one folder to another seems much faster.

A result!
Status: RESOLVED → VERIFIED
Blocks: 454829
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: