528568 - TB 3.0b4 crashes with high memory when working with large IMAP mailboxes

Status: RESOLVED WORKSFORME

(MailNews Core :: Backend, defect)

Description

[Thomas Harold]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4pre) Gecko/20090915 Thunderbird/3.0b4

Thunderbird will consistently crash due to running out of process memory when moving large quantities of messages from one folder to another.  In Windows Task Manager, you can watch the memory usage of TBird 3.0b4 climb to about 1.8-1.9GB of RAM before the TB window vanishes.

Reproducible: Always

Steps to Reproduce:
1. Find an IMAP folder with 3000 or 4000 or more messages
2. Select all or most, move them to a second folder on the IMAP server
3. Watch the memory usage spike in Task Manager
Actual Results:  
Thunderbird window vanishes.  No crash report gets submitted.

Expected Results:  
The messages should have been moved to the second IMAP folder (works fine in Thunderbird 2 release versions).

This should block Bug 515237 (beta 4 issues) and Bug 482756 (TB 3.x crashes without creating a bug report).

Comment 1

[[:Aureliano Buendía]]

Thomas can you provide your crash Id at http://crash-stats.mozilla.com/?

Comment 2

[Thomas Harold]

Thunderbird closes without creating a crash report at all.  No error message either from TB or Windows saying that "this app has encountered a problem".  The window will simply vanish once TB exhausts about 1.9GB of RAM.

Comment 3

[Wayne Mery (:wsmwk)]

This is not likely to be a regression in version 3. But is likely a dupe of one of several existing bugs about moving many messages - may be hard to say which one.

Thomas, were you running a specific test when you got this failure?  Or was this just "normal usage"?

Comment 4

[Thomas Harold]

Normal usage (reorganizing folders where I keep mailing list messages, such as PostgreSQL General).  Moving files between folders on an IMAP server, or between IMAP mailboxes results in a severe memory leak in beta 4.  I don't recall the issue being present in v3 beta 3 and was definitely not present in v3 beta 2.

In TB 2, I can easily move tens of thousands of messages between IMAP mailboxes.

Basically, watching Task Mgr, if I move a few hundred messages, TB memory usage balloons and never releases the memory until it is minimized for a while.  At which point it pegs the CPU and slowly releases some of the memory.  

No extensions / addons in use.

Comment 5

[Thomas Harold]

Bug 506509 might be related... on IMAP, deleting messages (regular delete) is essentially a move of the message from the original folder to the trash folder.

I do have mailnews.database.global.indexer.enabled enabled in this particular install of Thunderbird 3 beta 4.  Seems to be enabled by default.  However, even with turning this setting off and restarting TBird, moving ~4300 messages from one folder to another in the same IMAP account still results in the runaway memory issue.

Comment 6

[Thomas Harold]

Not fixed in the nightly build for RC1 (Thunderbird Setup 3.0 RC 1.exe	12-Nov-2009 22:44 / Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091112 Thunderbird/3.0)

Moving 2200 messages results in memory going from around 150MB up to 650-680MB.  Moving more then 4000 messages results in peak memory usage of over the limit of 1.9GB forcing Thunderbird to close.

Comment 7

[Wayne Mery (:wsmwk)]

bienvenu and/or asuth recently commented in another trunk regression iirc.

Comment 8

[WADA:World Anti-bad-Duping Agency]

Same problem as Bug 525646? See also Bug 519226 Comment #2.

Comment 9

[Thomas Harold]

Yes, it's possibly similar to Bug 519226 Comment #2, but I'm not sure that the rest of the bug is the same (especially not the original issue).  

Bug 525646 is very similar, as on IMAP, doing a delete is basically a move operation (TB moves the messages from the original folder to the "trash" folder).  However, given that 525646 was fixed on Nov 6th and I'm still seeing this in the nightly build from Nov 12th, I'm not sure that this is the same issue.

Comment 10

[Thomas Harold]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.3a1pre) Gecko/20091116 Shredder/3.1a1pre

This build (from the 1.9.3 line?) seems to address the issue.  Moving ~5000 messages between mailboxes no longer balloons the memory usage.

Comment 11

[Thomas Harold]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091117 Thunderbird/3.0

Not fixed in the 1.9.1 line as of RC 1 build 2.  Memory usage still balloons when moving messages between IMAP folders.  Moving ~1500 messages causes memory to go from 215MB up to 375MB (and moving even more messages in a single select would cause even higher memory usage).

Comment 12

[David :Bienvenu]

The increase in memory usage is significantly less in RC 1 than it was in b4, I believe.

Comment 13

[David :Bienvenu]

Thomas, are you using drag drop? On of the remaining issues has to do with drag drop, the other is still the large number of undo actions created for the move/copy. In any case, it's not an imap networking issue.

Comment 14

[Thomas Harold]

The original test case used drag-n-drop between IMAP folders, but Bug 528713 makes it currently impossible to test whether the issue still exists when using drag-n-drop to move the messages.

I'm redoing the test using the right-click, Move UI.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091112 Thunderbird/3.0
- moving ~15k messages using the right-click move UI results in memory usage bouncing between 700MB and 1.1GB.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4pre) Gecko/20090915 Thunderbird/3.0b4
- blows up due to memory ballooning out of control when moving a few thousand messages using the right-click move UI

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091117 Thunderbird/3.0
- memory usage bounces between 1.2GB and 1.9GB when moving ~15k messages using the right-click UI, stopping just short of where beta 4 would have crashed, but eventually overruns the 1.9GB limit and crashes

Comment 15

[Thomas Harold]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091117
Thunderbird/3.0
- Definitely still happens, just not as fast.  Moving messages by selecting in the message list with Ctrl-A, then right-click "Move to..." and pick another IMAP folder on the same account.  TB UI freezes, memory bounces around as it allocates/deallocates, but eventually it runs out of memory and crashes.  UI is completely unresponsive during this time.

The following message in the error log might be related:

Error: out of memory
Source File: file:///C:/Program%20Files/Mozilla%20Thunderbird%203%20RC%201%20build%202/modules/gloda/connotent.js
Line: 298

Comment 16

[Thomas Harold]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091121 Thunderbird/3.0
- Testing with build 3 of RC1.

IMAP folder with 11,900 messages.  Select all, right-click Move-To another folder in the same account.  Entire UI freezes up after starting the move process and stays frozen until TB finishes moving.  Memory usage bounces around, but TB does not run out of memory. Memory allocation / deallocation seems to be a lot better behaved in build 3.

I can still break it if gloda is indexing stuff in the background, and TB is downloading another large IMAP folder with a few thousand messages, and I move 20k messages from one IMAP folder to another.  TB tries to stay within the 1.9GB limit (memory used bounces around just shy of the limit) but eventually it trips across the limit and closes unexpectedly.

When it crashes, it's before it managed to move any messages at all into the destination folder.

I think my big complaint at this point with the move operation is the lack of UI responsiveness while TB is moving the messages.  On large moves of 20-30k messages, that means the UI locks up for 30 seconds or longer which is a bit unacceptable.  Especially annoying is if you immediately after the move finishes, the UI immediately locks up again for 10-30 seconds (maybe it's compressing the source folder).  

Since the lockup issue should probably be a new bug report, I'm going to go ahead and say that this should be marked as resolved.