Closed Bug 365838 Opened 18 years ago Closed 16 years ago

Deleting large number of mails in local mail subfolder hangs program with high memory usage

Categories

(Thunderbird :: Message Compose Window, defect)

Product:
Thunderbird
Component:
Message Compose Window
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: johan.plane, Unassigned)

Details

(Keywords: hang, perf)

User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) Build Identifier: 1.5.0.9 (20061207) When trying to delete 4502 messages stored in a subfolder, program stops executing transfer to trashcan at 4485 messages and stops responding. After forcing program out with the taskmanager, error reporting to Microsoft, upon restart of Thunderbird the subfolder content has to be rebuilt, however the message counter now shows the original 4502 messages PLUS the 4485 that was supposedly moved to trashcan. Reproducible: Always Steps to Reproduce: 1.Chose subfolder 2.Mark messages for deletion 3.chose delete button in menu, or hit delete button on keyboard Actual Results: Moving 4485 messages to trashcan, then halts before completion Expected Results: At least 4485 msgs in trashcan, however trashcan is empty and all messages are still in the subfolder they were supposedly deleted from
Is this a local folder or an imap folder? Do you have a virus checker installed that might be slowing down the copying of messages into the trash?
Version: unspecified → 1.5
It's a local folder. I have Norton AV 2006, but that doesn't check internal transfers I think. I have managed to get rid of the messages by deleting them a couple of hundred at a time, but the bug (in my opinion it is a bug) is still there.
I got the same problem trying to delete 1307 messages from an IMAP subfolder. I pushed Shift-Delete, so the messages are not to be copied to the trash folder. The program started by using 100% CPU and more and more memory. After a few minutes the messages were deleted, the CPU use returned to flat but the memory used was not returned to the system. Thunderbird is still using 1.4 GB. My test has been performed with version 2.0.0.0 (20070326).
(In reply to comment #0) > Windows NT 5.1; >(snip) > When trying to delete 4502 messages stored in a subfolder, program stops > executing transfer to trashcan at 4485 messages and stops responding. What is your PC's physical memory size? Probably long term swapping. Following is my test result using trunk on 2006/09/23. (Test) - My PC : 128MB RAM/Cerellon 600MHz notebook. - OS : MS Win-2K - Move 6000 mails from a folder to Junk by manual "Mark As Junk" - Average mail size=6KB (File size of a folder=39MB) (Result) It took 40 minutes. Maximum "Virtual Memory Size" while move = 90MB. (only 128MB RAM when my PC) Similar phenomenon to Comment #3 by Marcos Pirmez was observed, although the phenomenon continued 40 minutes when my environment. If you can test again, keep watching "Memory Size" and "Virtul Memory Size" and "CPU" for thunderbird.exe by Task Manager of MS Win, and wait for end of "Delete". (In reply to comment #3) > After a few minutes the messages were deleted, the CPU use returned to flat > but the memory used was not returned to the system. > Thunderbird is still using 1.4 GB. To Marcos Pirmez: "Memory Size" by Task Manger? Or "Virtual memory Size"? If you are talking about "Memory Size"(real memory size MS Win thinks allocated to thunderbird), try config.trim_on_minimize=true and minimize Thunderbird window after 1.4GB of "Memory Size". What will happen? To David Bienvenu : I think this is due to mismatch between Mozilla's memory use style and MS Win's memory management which is not so excellent for Tb.
Mark as Junk is very different from shift delete from an imap folder. For mark as junk, we have to tokenize each message to add it's token to the bad token list in the bayesian spam training set. Tokenizing each message means streaming it through the bayesian spam filter. If the tokens in those messages haven't been seen by the spam filter before, memory usage will increase to store those new tokens in the in -memory training set.
(In reply to comment #5) > Mark as Junk is very different from shift delete from an imap folder. Oh, sorry for my mis-leading test case, and thanks for response. When I tested "deleted of 6000 mails" according to bug report to Bugzilla Japan, critical situation(looks to be hang) was not observed even when my small 128MB PC. Phenomenon was slightly long execution time and relatively large memory usage and slightly large CPU usage only. "Mark As Junk" involves mail move(copy and delete) and "Mark As Junk" was convenient to force large memory usage and long term swapping on my PC using only 6000 small mails. So I used "Mark As Junk" to see phenomenon when severe swapping occurred. Please note that my question to reporter of IMAP shift+delete case is for not-released-real memory only.
To David: (Q1) On deleted/moved mail(original folder). "4502 messages PLUS the 4485" sounds to be phenomenon of Bug 363008. Is physical X-Mozilla-Status update of deleted/moved mail executed on each deleted/moved mail? Or after delete/move of all mails selected? (Q2) On destination folder of delete/move(Trash in this case) Possible cause of "No mail which had to be moved in Trash" is "extent was not updated due to kill of Tb and followed error". Can this situation occur?
x-mozilla-status is updated after *all* the messages have been copied, i.e., a move is a copy, followed by a delete, and a multiple message move is a multiple message copy, followed by a multiple message delete. I would expect that you would see the 4485 messages in the trash, and I would expect that all the original messages would be in the original folder, since the delete was basically aborted. But I don't really know what we were doing when the app was killed...
(In reply to comment #8) > x-mozilla-status is updated after *all* the messages have been copied, i.e., a > move is a copy, followed by a delete, and a multiple message move is a multiple > message copy, followed by a multiple message delete. Thanks for clear and easy to understand explanation. Following is quick observation of moving of 20344 mails with Tb 2.0.0. (1) During multiple copy phase, processed count in status bar is increased. (2) During the copy phase, file size and timestamp(last update) of target folder was updated. (checked by Windows explorer, View/Recent) (3) When processed mail count in status bar became near 20344(20200 or more), status bar update stopped(not increased any more), and no feed for a while. (looks to be hang for some people) (4) After a while, following move process(X-Mozilla-Status update) completes, and thread pane of original folder is cleared and count is set to zero. At same time, count of target folder is successufully increased(+20344). According to (3), status bar count update looks to be for each MM mails. If so, if total mail count is between N*MM and (N+1)*MM, last displayed count in status bar becomes N*MM. (Q1) David, is it true? > I would expect that you would see the 4485 messages in the trash (About reason of no count increase of Trash after kill&restart) If (Q1) is true, killing of Tb by opener was done between (3) and (4). Another evidence of this is "all mails were kept in original folder". Since directory update looks to be done correctly on each(or several) adding(s) of extent, I can imagine other reason than "no rebuild index" by time-stamp play(5 minutes?) in decision of re-build index during restart. (I experienced "no re-building of msf" several times when I modified mail ) ( file content quickly and restarted Thunderbird very shortly. ) (Q2) David, can this occur?
Correction: I can imagine => I can't imagine. Sorry for spam.
The not rebuilding index issue is weird - yes, we ignore the timestamp more now, but we pay attention to the file size - if it's ever different from what we expect, we rebuild the index. And if you've copied a bunch of messages into a folder, the size is going to be different... Re the count, we might not update the count for the last block of messages copied, if that's what you're seeing.
(In reply to comment #11) > we ignore the timestamp more now, but we pay attention to the file size > - if it's ever different from what we expect, we rebuild the index. To David Bienvenu : Is it true when Tb 1.5.0.9? Bug opener's case is when Tb 1.5, not Tb 2.0/trunk.
(In reply to comment #0) To Johan Plane(bug opener) : > When trying to delete 4502 messages stored in a subfolder, program stops > executing transfer to trashcan at 4485 messages and stops responding. How long did the "stops responding" continue? > After forcing program out with the taskmanager When did you kill Tb? (how many minutes did you watch "stops responding" before killing"?) See my observation of step (3) in Comment #9.
Another question to Johan Plane(bug opener) : Do you enable option setting which clears Trash automatically? (for example, "Empty Trash on exit" of Account Settings/Server Settings)
Yes it is enabled and I have not tried with an alternate setting.
(In reply to comment #15) > Yes it is enabled Thanks for quick responding. Johan, what is your answer to my question of Comment #13?
(In reply to comment #13) Approx 60 minutes. On a P4 dual core with 1024 Mb RAM. > (In reply to comment #0) > To Johan Plane(bug opener) : > > When trying to delete 4502 messages stored in a subfolder, program stops > > executing transfer to trashcan at 4485 messages and stops responding. > How long did the "stops responding" continue? > > After forcing program out with the taskmanager > When did you kill Tb? (how many minutes did you watch "stops responding" before > killing"?) > See my observation of step (3) in Comment #9.
Johan, do you have any virus checker running that could be interfering with or slowing down the file operations involved in deleting a bunch of messages?
I ran Norton 2006, now 2007 and it does not check internal transfers. Mail is only checked for viruses and worms upon arrival or when trying to save an attachment, not when moving mail.
Not sure what you mean by "internal transfers" but deleting a bunch of messages involves reading them from one file, and writing them to an other file (the trash), and then updating/writing the original folder to update the x-mozilla-status headers for all the messages that were deleted.
Norton guards incoming and outgoing mail ports, aswell as saving attachments. It does NOT check mails being copied from one folder to another.
(In reply to comment #17) > Approx 60 minutes. On a P4 dual core with 1024 Mb RAM. I see. (Q1) When "hang for 60 minutes", (Q1-A) does virual/real memory use by Thunderbird increase rapidly?, (Q1-B) is almost all 1024 MB of real memory used by TB? If (Q1-B) is yes, severe swapping is observed? (Both "Virtual Memory Size" column and "Memory Usage" column of thunderbird.exe in "Process Tab" of MS Win's Task Manager)
(In reply to comment #21) > Norton guards incoming and outgoing mail ports, as well as saving attachments. > It does NOT check mails being copied from one folder to another. It sounds that "as well as saving attachments" means on access file scan(file scan on file creation/modification) is enabled. Does your "does NOT check mails being copied" mean you excluded Tb's mail directory from target directory of on access file scan?
Norton checks the mail once only and that is when it is INCOMING. Moving (even if it incorporates copying) is not monitored. As any attachments are embedded with the message in TB, the only other time Norton checks is when an attachment is being saved separately to disk, i.e. extracted from the embedded TB messagefile.
(In reply to comment #22) > (In reply to comment #17) > > Approx 60 minutes. On a P4 dual core with 1024 Mb RAM. > I see. > (Q1) When "hang for 60 minutes", (Q1-A) does virual/real memory use by > Thunderbird increase rapidly?, (Q1-B) is almost all 1024 MB of real memory used > by TB? If (Q1-B) is yes, severe swapping is observed? > (Both "Virtual Memory Size" column and "Memory Usage" column of thunderbird.exe > in "Process Tab" of MS Win's Task Manager) Q1A & B Memory usage increases yes, but not to any alarming levels, always has at least 200MB free of the 1024 available. Swap file size increases marginally. When TB hangs, CPU usage by TB is down to 0 (zero).
(In reply to comment #24) > Norton checks the mail once only and that is when it is INCOMING. Moving (even > if it incorporates copying) is not monitored. But, as I said in Comment #23, you looks to enable on access file scan(file scan on file creation/modification), so I can't believe "Moving (even if it incorporates copying) is not monitored" in your environment. If you don't exclude Tb's mail directory and/or if you don't limited "on access scan" for pre-defined file extension(please note that Tb's mail folder file is file which has no file extension), your anti-virus software will always does do file scan of Tb's mail file folder, isn't it? Please note that "mail folder file update (both adding mail data and changing X-Mozilla-Staus:/X-Mozilla-Status2: header data update) is file modification. What is your exact setting for your anti-virus software?
"file scan on file creation/modification" is not a variable in Norton 2007! "your anti-virus software will always does do file scan of Tb's mail file folder, isn't it?" Only if it is included in a general, scheduled, scan of the entire computer! But that is a manual or manually induced scan! I beleive that the AV software settings are unimportant to the issue in queston, and I will therefore not comment further on anything related to this.
I don't see this with trunk version 3.0a1pre (2007071705), symantec AV enterprise 10.1.6. But I can't say my version of AV is comparable to Johan's SAV 2007 with write caching ENABLED for the disk Maxtor 6Y160M0 (7200 rpm SATA) 10,370 messages (resulting folder size 72MB) 20 seconds to copy from local folder A to local folder B. 1:40 to delete from folder B, 40seconds of that with status at 10,3xx messages copied subset test of 6,000 messages with disk write caching DISABLED ~20 seconds to copy 1:10 to delete from folder B, 40seconds of that with status at 6,940 messages copied Memory usage nominal (not more than normal) and speed quite acceptable. So either: 1. my PC+disk is much faster (P4 3.2Ghz, 1gig memory) 2. something on Johan's PC is slowing it down AND causing high memory usage 3. thunderbird trunk is much better As for #1, only if PC is 10x slower than mine would the performance be so bad. But I did not see the memory problem => no evidence of this problem on trunk. Likewise for comment 4 above - wada tested on dinky notebook which probably had a 3600rpm disk, so not surprising his test took 40 minutes. Moving on to #2 and #3 - a quick and simple test will prove whether SAV is the issue - turn off SAV and do a small testcase. Once AV is definitely eliminated as the cause, you might try a trunk build or look for some other cause on your PC. Let us know what happens. Or if you don't wish further assistance, please close the bug INCOMPLETE. If your PC is on a UPS and you feel safe, you might want to turn on write caching and give that a try.
Summary: Deleting large number of mails in subfolder hangs program → Deleting large number of mails in local mail subfolder hangs program with high memory usage
I have tried again. Before the test thunderbird was using 40MB (according to task manager). I clicked on a folder with 1939 unread messages, then I typed control-A, then shift-del. The CPU use raised to 100%. After 1 minute, the memory used by thunderbird was 375MB. After 10 min, it was 1624MB. 13 minutes after my shift-del, when the memory usage was about 1700 MB, the program simply stoped execution. During all the 13 minutes there where negligible network and disk activity. Just processor activity. When I restarted the program, thunderbird reported 1038 messages on the mailbox. When i clicked again on the mailbox 901 headers where read from the server and thunderbird reported 1939 messages. The configuration used for this test: CPU amd athlon 64 x2 dual, 2.81GHz, 2GB ram Windows XP SP2 No anti virus Access to the mail server via IMAP Thunderbird version 2.0.0.6 (20070728) Folder name INBOX/RecorteDJ/Copia
(In reply to comment #29) > Windows XP SP2 > After 10 min, it was 1624MB. Number displayed in which column of Process Tab of Task Manger of MS Win? "Virtual Memory Size"? Or "Memory Usage"? To Marcos Pirmez: Please distinguish "Memory Usage" column and "Virtual Memory Size" column. For large number itself in "Memory Usage" column on MS Win, read Bug 381950 and documents pointed in it, please. > Before the test thunderbird was using 40MB (according to task manager). > I clicked on a folder with 1939 unread messages, then I typed control-A, then shift-del. > The CPU use raised to 100%. > After 1 minute, the memory used by thunderbird was 375MB. > After 10 min, it was 1624MB. 13 minutes after my shift-del, > when the memory usage was about 1700 MB, the program simply stoped execution. > (snip) mail server via IMAP > (snip) 2GB ram Sounds "Memory Usage" column(real memory size that MS Win thinks Tb is using). How about "Virtual Memory Size" column increase? Same phenomenon is observed even when -safe-mode of Thunderbird? To Marcos Pirmez: Anyway, IMAP case involves too many elements compared to local mail folder case - server connection, server access(FETCH,APPEND,DELETE...), folder re-syncronization etc. I think you'd better to open separate bug for phenomenon of large number of "Memory Usage" on MS Win(then thrashing finally) when IMAP multi-mail delete. Please attach Performance Counter log while above test(all of Process Counter for thunderbird.exe) to the bug.
I got the same problem with only some folders. Even when I try to delete 74 of 907 messages it Tb seems to hang. It needs more than ten minutes to finish the delete. Viruses only are checked when coming in.
I'm seeing the same. If I try to delete several thousand emails, Thunderbird hangs "indefinately". On the email server (Postfix v2.1.5, Courier-IMAP v3.0.8), I see that an increasing amount of imap processes is spawned. It seems like the server takes some time to respond, which Thunderbird interprets as a timeout, and retries the delete operation (again and again). The last time I tried deleting 4000 emails from Junk, I had to force quit Thunderbird. When I looked into the Trash folder, it had 5 (five) copies of each Junk mail (i.e. 20000 emails). The Junk mail folder was untouched. We are several people here experiencing this, on different platforms. MaxOSX 10.5, 2GB RAM, Thunderbird 2.0.0.9, IMAP.
Keywords: hang
I am experiencing this with thunderbird 2.0.0.14 on windows xp2 sp3 whenever I delete emails from any folder if it is more than about 30 emails there is a delay the progress counter increases quickly to something more than 90% of the no. of files and then stops. The delay increases with the number of emails to delete and with large numbers ( 3 or 4 hundred or more) the delay is so long it is effectively a hang. when it does resolve the emails have been deleted successfully and can be found in the trash folder. E.g. 20 emails took 100% cpu for 10 secs to complete a delete. (1.4ghz cpu 1gb ram)
By Bug 429753, following was found. When long thread exists, .msf update upon delete of many mails takes long. To bug opener and all problem reporters: Is very long(or deep) thread involved in your problem? (many mails have same subject, or many mails have reply-to: like news post) Because .msf update related issue, deleted but not-expunged-yet mails possibly make situation worse. Do you execute "Compact Folder"(and "Empty Trash") regularly?
No I don't compact folders because I empty them regularly, however I do empty the trash folder very regularly because I don't like dodgy emails hanging around! Thanks for your quick response.
If you only "empty them" (== delete the mails) it only means the mails get "marked" as deleted. Compact then removes them physically from the mail file. So compacting folders regularly might help with the problem...
Assignee: mscott → nobody
Thanks for that advice - I will assume that compacting will reduce the problem as have never done that!!!! - Is there a reason why it is implemented in this way? It seems to me that if I want to delete mail then it should go to trash and when I empty trash they should be gone!
Having compacted all folders I have now tried to delete 90 emails from a sub folder of inbox - oh dear! still gets to about 87 emails on the progress bar and then with the cooling fan whirring at top speed it sits there for 10 seconds. - same problem.
Dup. of bug 296453?
(In reply to comment #39) > Dup. of bug 296453? quite likely - but possibly not for reporter, who states "Q1A & B Memory usage increases yes, but not to any alarming levels, always has at least 200MB free of the 1024 available. Swap file size increases marginally. When TB hangs, CPU usage by TB is down to 0 (zero)." Johan, can you reproduce this problem using a current nightly build? backup your profile first, and install using "custom" to a new directory ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-trunk/ ppl should check a nightly build once bug 296453 makes progress.
Component: General → Message Compose Window
Keywords: perf
QA Contact: general → message-compose
(In reply to comment #38) > I have now tried to delete 90 emails from a sub folder of inbox > - oh dear! still gets to about 87 emails on the progress bar It's Bug 388527. > and then with the cooling fan whirring at top speed it sits there for 10 seconds. Probably similar phenomenon to Bug 452221, which is spin-off of a specific issue in Bug 296453. See Bug 452221 Comment #4. Delete(move to Trash) roughly consist of; (1) Copy mails to Trash folder. (2) Add "deleted flag" to X-Mozilla-Keys: header of mails in source folder (Count is probably displayed at step (1) or (2). Bug 388527 is observed.) (3) ".msf" update of source folder Shift+Delete(tested case by Bug 452221) is (2)+(3), so phenomenon of Bug 452221 occurs at above step (2)/(3) if some conditions are met.
This has been vastly improved in TB 2 even if it hasn't dissappeared alltogether. However, I do not intend to persue this further and therefore close the issue.
Status: UNCONFIRMED → RESOLVED
Closed: 16 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.