TB 3.0b4 crashes with high memory when working with large IMAP mailboxes

RESOLVED WORKSFORME

Status

MailNews Core
Backend
--
critical
RESOLVED WORKSFORME
8 years ago
8 years ago

People

(Reporter: Thomas Harold, Unassigned)

Tracking

(Blocks: 1 bug, {crash, perf, regression})

Trunk
x86
Windows XP
crash, perf, regression

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

8 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4pre) Gecko/20090915 Thunderbird/3.0b4

Thunderbird will consistently crash due to running out of process memory when moving large quantities of messages from one folder to another.  In Windows Task Manager, you can watch the memory usage of TBird 3.0b4 climb to about 1.8-1.9GB of RAM before the TB window vanishes.

Reproducible: Always

Steps to Reproduce:
1. Find an IMAP folder with 3000 or 4000 or more messages
2. Select all or most, move them to a second folder on the IMAP server
3. Watch the memory usage spike in Task Manager
Actual Results:  
Thunderbird window vanishes.  No crash report gets submitted.

Expected Results:  
The messages should have been moved to the second IMAP folder (works fine in Thunderbird 2 release versions).

This should block Bug 515237 (beta 4 issues) and Bug 482756 (TB 3.x crashes without creating a bug report).
Thomas can you provide your crash Id at http://crash-stats.mozilla.com/?
Severity: major → critical
Keywords: crash
(Reporter)

Comment 2

8 years ago
Thunderbird closes without creating a crash report at all.  No error message either from TB or Windows saying that "this app has encountered a problem".  The window will simply vanish once TB exhausts about 1.9GB of RAM.

Comment 3

8 years ago
This is not likely to be a regression in version 3. But is likely a dupe of one of several existing bugs about moving many messages - may be hard to say which one.

Thomas, were you running a specific test when you got this failure?  Or was this just "normal usage"?
Blocks: 482756
Summary: TB 3.0b4 crashes when working with large IMAP mailboxes → TB 3.0b4 crashes with high memory when working with large IMAP mailboxes
(Reporter)

Comment 4

8 years ago
Normal usage (reorganizing folders where I keep mailing list messages, such as PostgreSQL General).  Moving files between folders on an IMAP server, or between IMAP mailboxes results in a severe memory leak in beta 4.  I don't recall the issue being present in v3 beta 3 and was definitely not present in v3 beta 2.

In TB 2, I can easily move tens of thousands of messages between IMAP mailboxes.

Basically, watching Task Mgr, if I move a few hundred messages, TB memory usage balloons and never releases the memory until it is minimized for a while.  At which point it pegs the CPU and slowly releases some of the memory.  

No extensions / addons in use.
(Reporter)

Comment 5

8 years ago
Bug 506509 might be related... on IMAP, deleting messages (regular delete) is essentially a move of the message from the original folder to the trash folder.

I do have mailnews.database.global.indexer.enabled enabled in this particular install of Thunderbird 3 beta 4.  Seems to be enabled by default.  However, even with turning this setting off and restarting TBird, moving ~4300 messages from one folder to another in the same IMAP account still results in the runaway memory issue.
(Reporter)

Comment 6

8 years ago
Not fixed in the nightly build for RC1 (Thunderbird Setup 3.0 RC 1.exe	12-Nov-2009 22:44 / Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091112 Thunderbird/3.0)

Moving 2200 messages results in memory going from around 150MB up to 650-680MB.  Moving more then 4000 messages results in peak memory usage of over the limit of 1.9GB forcing Thunderbird to close.

Comment 7

8 years ago
bienvenu and/or asuth recently commented in another trunk regression iirc.
Component: General → Folder and Message Lists
Keywords: regression
QA Contact: general → folders-message-lists
Version: unspecified → 3.0
Same problem as Bug 525646? See also Bug 519226 Comment #2.
(Reporter)

Comment 9

8 years ago
Yes, it's possibly similar to Bug 519226 Comment #2, but I'm not sure that the rest of the bug is the same (especially not the original issue).  

Bug 525646 is very similar, as on IMAP, doing a delete is basically a move operation (TB moves the messages from the original folder to the "trash" folder).  However, given that 525646 was fixed on Nov 6th and I'm still seeing this in the nightly build from Nov 12th, I'm not sure that this is the same issue.
(Reporter)

Comment 10

8 years ago
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.3a1pre) Gecko/20091116 Shredder/3.1a1pre

This build (from the 1.9.3 line?) seems to address the issue.  Moving ~5000 messages between mailboxes no longer balloons the memory usage.
(Reporter)

Comment 11

8 years ago
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091117 Thunderbird/3.0

Not fixed in the 1.9.1 line as of RC 1 build 2.  Memory usage still balloons when moving messages between IMAP folders.  Moving ~1500 messages causes memory to go from 215MB up to 375MB (and moving even more messages in a single select would cause even higher memory usage).

Comment 12

8 years ago
The increase in memory usage is significantly less in RC 1 than it was in b4, I believe.
Component: Folder and Message Lists → Networking: IMAP
Keywords: perf
Product: Thunderbird → MailNews Core
QA Contact: folders-message-lists → networking.imap
Version: 3.0 → Trunk

Comment 13

8 years ago
Thomas, are you using drag drop? On of the remaining issues has to do with drag drop, the other is still the large number of undo actions created for the move/copy. In any case, it's not an imap networking issue.
Component: Networking: IMAP → Backend
QA Contact: networking.imap → backend
(Reporter)

Comment 14

8 years ago
The original test case used drag-n-drop between IMAP folders, but Bug 528713 makes it currently impossible to test whether the issue still exists when using drag-n-drop to move the messages.

I'm redoing the test using the right-click, Move UI.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091112 Thunderbird/3.0
- moving ~15k messages using the right-click move UI results in memory usage bouncing between 700MB and 1.1GB.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4pre) Gecko/20090915 Thunderbird/3.0b4
- blows up due to memory ballooning out of control when moving a few thousand messages using the right-click move UI

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091117 Thunderbird/3.0
- memory usage bounces between 1.2GB and 1.9GB when moving ~15k messages using the right-click UI, stopping just short of where beta 4 would have crashed, but eventually overruns the 1.9GB limit and crashes
Component: Backend → General
Product: MailNews Core → Thunderbird
Version: Trunk → 3.0
(Reporter)

Comment 15

8 years ago
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091117
Thunderbird/3.0
- Definitely still happens, just not as fast.  Moving messages by selecting in the message list with Ctrl-A, then right-click "Move to..." and pick another IMAP folder on the same account.  TB UI freezes, memory bounces around as it allocates/deallocates, but eventually it runs out of memory and crashes.  UI is completely unresponsive during this time.

The following message in the error log might be related:

Error: out of memory
Source File: file:///C:/Program%20Files/Mozilla%20Thunderbird%203%20RC%201%20build%202/modules/gloda/connotent.js
Line: 298
Component: General → Account Manager
Version: 3.0 → 0.9
Component: Account Manager → Backend
Product: Thunderbird → MailNews Core
Version: 0.9 → Trunk
(Reporter)

Comment 16

8 years ago
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091121 Thunderbird/3.0
- Testing with build 3 of RC1.

IMAP folder with 11,900 messages.  Select all, right-click Move-To another folder in the same account.  Entire UI freezes up after starting the move process and stays frozen until TB finishes moving.  Memory usage bounces around, but TB does not run out of memory. Memory allocation / deallocation seems to be a lot better behaved in build 3.

I can still break it if gloda is indexing stuff in the background, and TB is downloading another large IMAP folder with a few thousand messages, and I move 20k messages from one IMAP folder to another.  TB tries to stay within the 1.9GB limit (memory used bounces around just shy of the limit) but eventually it trips across the limit and closes unexpectedly.

When it crashes, it's before it managed to move any messages at all into the destination folder.

I think my big complaint at this point with the move operation is the lack of UI responsiveness while TB is moving the messages.  On large moves of 20-30k messages, that means the UI locks up for 30 seconds or longer which is a bit unacceptable.  Especially annoying is if you immediately after the move finishes, the UI immediately locks up again for 10-30 seconds (maybe it's compressing the source folder).  

Since the lockup issue should probably be a new bug report, I'm going to go ahead and say that this should be marked as resolved.
Status: UNCONFIRMED → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED

Updated

8 years ago
Resolution: FIXED → WORKSFORME
You need to log in before you can comment on or make changes to this bug.