583365 - Memory leak ending in OOM crash moving 70k emails from inbox to archive (memory for bulk mail move/delete, and memory from "select+all and move/delete" is not released until TB window closed and opened)

Status: NEW

(Thunderbird :: General, defect)

Description

[Jacob Taylor]

User-Agent:       Opera/9.80 (Windows NT 6.1; U; en) Presto/2.6.30 Version/10.60
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.7) Gecko/20100713 Thunderbird/3.1.1

I attempted to move all 70k of my emails from my inbox into the `yearly` archives by selecting my inbox(ctrl+a) and hitting the archive button. After slowly ballooning to using over 1.5 gigs of ram (I only have 2 gigs of ram) Thunderbird crashed. From what I can tell after relaunching Thunderbird, it managed to move 2010 and part of 2008 before crashing (none of 2009,07,06 or 05). Something around 25k emails.

Reproducible: Didn't try

Steps to Reproduce:
1.Select all of your inbox with many emails (ctrl+a or similar)
2.Hit the archive button
3.Watch your available ram sink away and Thunderbird crash within minutes.


Expected Results:  
Moved all the emails into the archive with some fuss, but not crashing the app.

Comment 1

[WADA:World Anti-bad-Duping Agency]

> Thunderbird crash within minutes.

AFAIK, MS Windows automatically expands page file by default, if real memory shortage occurs. So, phenomenon is usually "thrashing" like one or "system freeze" like one. Do you set some size limitations of allocated real memory or virtual memory to an application?

> After slowly ballooning to using over 1.5 gigs of ram (snip)

"What value at where" do you say by "1.5gig"?

I could see Tb's large Virtual Memory size("Virtaul Memory Size" of MS Win-XP's Task Manager, "Private Bytes" of Process Explorer) and large Real Memory size("Mem Usage" of MS Win-XP's Task Manager, "Working Set" of Process Explorer).
Quick check result of Archive, copy&Shift+Delete with Tb 3.1.1 on MS Win-XP.
(0) Gloda(Global Indexer) is off.
    80,000 mails(1KB each) in a local mail folder say (Test)
    All mail has different subject, no Reply-To: header etc.
    All mail has Date: of 1970/mm/dd
    Archive option=default(archived to Archived/1970 in this case).
(1) Select+All of 80,000 mails at Test, Archive => moved to Archived/1970
    => Took long, 50%CPU of core2duo while delete step, but normally ended.
    => Real Memory size of Tb : around 500 MB (1GB memory on PC)
       (sorry but I didn't check VM size at this step)
(2) Select+All of 80,000 mails at Archived/1970, Copy to Test
    => Copy ends normally, took long though.
(3) Shift+Delete at Archived/1970
    => Real    memory size increased to near 900MB
       Virtual memory size increased to 900MB(near 1GB)
    => "Unresponsive script" was issued repeatedly for several scripts.
    => It took very long, but Shift+Delete ended normally.
    => Real Memory size/Virtual Memory size is same even after end of delete,
       even after compact of Archives/1970 and test, folder switch etc.
(4) Open new Tb window via "Open" of account at folder pane,
    and close old Tb window.
    => Real memory size    of Tb declined to  70 MB
    => Virtual memory size of Tb declined to 740 MB
    Probably garbage collector runs.
    Resources are probably referred by someone of an Tb window at step 3,
    even after resources are not used any more.
    Remained VM size=740MB may be memory leak, but may be merely fragmentation.
(5) Open new Tb window via "Open" of account at folder pane,
    and close old Tb window.
    => Real memory size/Virtual memory size of Tb is not changed any more.
"Bulk move" is better to be improved when move of many many mails, in order not use excess memory.

If long or deep thread exists, Bug 452221 occurs, and peak virtual memory size increases(peak real memory size too according to VM size increase), because garbagge collection doesn't run for long time due to too heavy CPU usage.
If Gloda(Global Indexer) is enaled, Gloda also uses Virtual memory(and real memory too), because Gloda needs to update Gloda's DB according to mail move.

Does long or deep thread exist in your 70K mails?
Do you enable Gloda?
Local mail folder(POP3, Local Folders)? Or IMAP folder? Or both?

Comment 2

[Ludovic Hirlimann [:Usul]]

Besides answering Wada could you also give us a crash ID please ? (see http://support.mozillamessaging.com/en-US/kb/Mozilla%20Crash%20Reporter)

Comment 3

[Jacob Taylor]

(In reply to comment #1)
> AFAIK, MS Windows automatically expands page file by default, if real memory
> shortage occurs. So, phenomenon is usually "thrashing" like one or "system
> freeze" like one. Do you set some size limitations of allocated real memory or
> virtual memory to an application?
No, I don't set limitations, and my swap file does not change size dynamically. It is a fixed size, both minimum and maximum size are equal.

> "What value at where" do you say by "1.5gig"?
Task manager calls the number `Memory: Private Working Set`


> (0) Gloda(Global Indexer) is off.
My global index is on.

> Does long or deep thread exist in your 70K mails?
Yes, many. I'm unsure as to what you consider a long thread, but it's not unheard of to have a thread in excess of 70 replies

> Local mail folder(POP3, Local Folders)? Or IMAP folder? Or both?
Only pop3, no IMAP.

@Ludovic The crash reporter didn't give me a crash ID (or I didn't know to keep it if it did). It started when TB finally crashed, but said there was a problem submitting the report. That's why I submitted this manually.

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #3)
> No, I don't set limitations, and my swap file does not change size dynamically.
> It is a fixed size, both minimum and maximum size are equal.

Aha. I've understood why crash occurred in your environment. Tb was killed by OS because Tb requested too much virtul memory.

> > "What value at where" do you say by "1.5gig"?
> Task manager calls the number `Memory: Private Working Set`

Real memory size related column? Or Virtual memory size related column?
As seen in my test, real/virtual is not so important for your crash in comment #0(not so big difference between peak real size and peak virtual size). But, as problem is memory leak releated(aparenat remained reference to non-needed resource was observed in my test), Real Memory size or Virtual Memory size is important. Please check both value in next test. Getting Performance Counter log is better for checking memory size of Tb. 

> > (0) Gloda(Global Indexer) is off.
> My global index is on.

Can you check with Gloda(Global Indexer)=Off, with same or similar 70K mails or smaller number of mails again?
It may force re-indexing when Gloda is enabled again. If you have huge volume of mail, check with new profile and single mail folder of the 70K mails is recommended.
If crash won't occur like my test, check "Open new Tb window followed by close of old Tb window", please.

Comment 5

[WADA:World Anti-bad-Duping Agency]

Confirming by my test result of comment #1.

Comment 6

[WADA:World Anti-bad-Duping Agency]

Crash itself is due to smaller swap file size than required. Changing to major.

Comment 7

[Jacob Taylor]

I just crashed Thunderbird again attempting a similar operation. I tried to unarchive all my mail (to see if I could make it crash again and get a crash dump and ID), and it put it all in the wrong account's inbox (I have two gmail accounts, each with an archive sub-folder). However, when I tried to bulk move the mail to the correct inbox, it crashed in a similar way as before. I don't know if ram is the issue this time, as I've recently upgraded to 4 gigs of ram. 

This time, I got an error window that says 
title -> "Microsoft visual c++ runtime library"
"Runtime error!"
"file location of thunderbird"
"The application has requested the runtime terminate it in an unusual way."
task manager says "1,699,400K" ram used in "Memory, Private working set"
http://img534.imageshack.us/img534/4346/processxp.png is the information I can see. I've got full blown win7 and more ram now, so it seems the bug persists... I'm thinking of wiping out thunderbird, redownloading all my mail and trying to replicate... would that be helpful?

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #7)
> I tried to unarchive all my mail, and it put it all in the wrong account's inbox
> (I have two gmail accounts, each with an archive sub-folder).

What operation do you call "unarchive"?
Merely you moved "all mails in Archives of accountA" to "Archives of accountB"?
If yes, the "unarchive" is also a bulk move of many mails. It leaks memory as I wrote in comment #1, even if crash due to swap size limitation doesn't occur. So, memory leak by next "archive"("bulk move of many mails" too) may produce crash due to swap size limitation.  
  
No need of wiping out current Thunderbird environment(profile) and redownloading all mails. Check with new profile(no add-on, ...) is sufficient.
1. Create a new Tb profile, terminate Tb, restart Tb.
   => only "Local Folders" is created.
   Create mail folders/subfolders required for testing.
   Disable Global Search and Indexer.
2. Copy mail folder file(s) of current profile to ...\Mail\Local Folders.
   If Inbox folder, copy file of Inbox(not Inbox.msf) to new profile.
   If folder Z under A, copy file of A.sbd\Z to new profile.
3. Check Tb's behaviour upon move/delete of 70K mails in "Local Folders".

Do you see phenomenon like my comment #1?
Does Tb's crash still occur by "move of 70K mails after restart of Tb"?
If crashed due to swap size limitation, check with 60K mails, 50K mails, ..., please. It's not test of "crash due to swap size limitation after severe memory leak of Tb". It's test for "memory leak of Tb upon bulk move of many mails".

Comment 9

[WADA:World Anti-bad-Duping Agency]

Two kinds of memory leak seems involved in my test result of comment #1.
  - resource for undo delete(bug 538378)
  - resource not for undo delete(bug 398684)

Comment 13

[Michelle]

Is this where I write how TB (since vs 91 I think) is taking up so much of my memory?

I have a feeling it's b/c you guys changed it so now there's multiple windows open for every e-mail or even search box.

When I close an e-mail that's in draft mode, my memory goes down.

Why did you change it so that there's multiple windows open now?

Thanks