387998 - when many moving messages from one folder to another cpu is at 100% and process taking huge amounts of time

Status: RESOLVED DUPLICATE of bug 296453

(Thunderbird :: Message Compose Window, defect)

Description

[Seceleanu Cristian]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4
Build Identifier: version 2.0.0.0 (20070326)

Hi, i'm using Linux and thunderbird as pop3 client.
I'm a system admin and daily i get over 3k e-mail messages in my inbox and from time to time i need to move them to other folders (as when they accumulte intro one folder it works much much slower). 
The problems is that when i move say 10000 messages from a folther with 100000 messages it take some 2h to complete the operation and it eats over 300MB RAM to do it and of course cpu is at 100% usage.
This is an optimization issue, not a bug, and i'm sure you can fix it provided you are interested in improving this excellent e-mail client.

Reproducible: Always

Steps to Reproduce:
1.Fill a folder with more than 100k messages
2.Create another folder or subfolder, it does not matter
3.Move 10000 messages from the 100k messages folder to the newly created one

Actual Results:  
It took over 2h time to complete, during which it used 100% CPU and 300MB RAM

Expected Results:  
Finish in less than 10 min with less resources

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

When you are getting such an amount of messages day by day have you ever compacted your folders? It's only a vague feeling but if you have never did that please try it now: http://kb.mozillazine.org/Compacting_folders

Does it solve your issue? If not, where is your destination folder located? Locally? Another account? IMAP account?

Comment 2

[Seceleanu Cristian]

I compact my folders some 5 - 10 times a day as it also takes some 5 minutes each time and i don't want to do it too many times if you understand what i mean....

I have 5 accounts, 3 have separate folders, 2 use the local folders, i'm experiencing this with all of them (i have one with 300k email messages, another with some 180k and a couple more folders with 20k )

Deleting the older messages is not an option for me as i need to keep them for many reasons. Don't get me wrong, i tried using outlook :) and it dies when i hit the 32k limit in one folder :) so it's far better using thunderbird but still i think that it can be improved regarding this issue 

PS. All 5 are pop3 accounts.

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> it take some 2h to complete the operation and 
> it eats over 300MB RAM

WORKSFORME with Tb 2.0.0.x/Seamonkey 1.1.x/many Tb trunks builds on my Win-XP SP2 PC.

To Seceleanu Cristian(bug opener):

(Q1) Which do you mean by "300MB RAM", real memory used by Tb, or virtual memory used by Tb?
(Q2) When "real memory used by Tb", how many physical memory do you have?
Was heavy swapping observed while mail moving?
(Q3) When "virtual memory used by Tb", do you mean that very large VM size increase was observed after start of move mails/while moving mails?.
(Q4) Does your problem occur even with newly created profile?
(Q5) Do you use aniti-virus software? If yes, will "excluding of mail folder files from scan target" alter phenomenon? Try next.
 1. Say source folder="AA" and target folder="BB"
 2. Esxclude "AA", "AA.msf", "BB" and "BB.msf" from scan target
 3. Move many mails from AA to BB
(Q5) Do you use "Google Desktop"?

Comment 4

[Seceleanu Cristian]

Hi,

A1 to Q1:
300MB real RAM
A2 to Q2:
512 MB RAM
Heavy swapping up to 800 MB swap sometimes (i tried once to move 30k messages in one move ) and i notice something else...on normal usage, not moving e-mails, just opening folders, reading and receiving e-mail i notice some 250 - 300 M swap usage  and about 180 - 200M RAM usage (after i close thunderbird i notice more free RAM with about 200 M and swap returns to 0  after being 250 - 300 MB used)
A3 to Q3:
yes
A4 to Q4:
yes, after a couple of days of e-mail downloading when i get all e-mail in my inbox it does the same thing
A5 to Q5:
No antivrirus, i am using linux, there is no need for that here and no i do not use google desktop

Also i would like to mention that in total my thunderbird profile uses 4.6 Gbytes of disk space (99.9% of disk space are most likely my e-mail messages) -> this is probably the cause for A2 above but still even in my case i think that it could be optimized to eat less resources... 

I'd like to mention that i am willing to help fix this bug either by writing code or providing some ideas to thunderbird developers so that the fix can be included in trunk asap.

Thanks

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #4)
> A4 to Q4:
> yes, after a couple of days of e-mail downloading when i get all e-mail in my
> inbox it does the same thing

My "new profile" implies "no extension/add-on", and it doesn't mean "newly created but customized".
(Q7) Will your problem be re-produced by following test?
1. Create new Thunderbird profile.
2. Define a POP3 account(dummy server, No Global Inbox).
3. Terminate Thunderbird
4. Copy file of "Inbox"(not "Inbox.msf") in Mail directory, which is you are
   using daily, to Mail directory of the newly created profile.
5. Restart Thunderbird, and click "Inbox", and confirm "Inbox" is healthy.
6. Define a mail folder (say "Test")
7. Move many mails from "Inbox" to "Test"

Comment 6

[Seceleanu Cristian]

Hi,

I will perform this test but unfortunately i have to leave town for about 2 weeks. I will let you know how this test turned out when i get back.

Also i'd like to mention that i have done something similar to Q7 before switching entirely to linux, i backed up my profile from windows then after switching to linux i used thunderbird --profilemanager and pointed to the folders from windows ... it worked ok, i mean folders were healthy, i could see all e-mail messages, i compacted the folders just in case, but when moving e-mail messages from one folder to another the same problem occured.

...i think i forgot to mention that this happened on both windows and linux versions of thunderbird

Another thing i noticed is: actual disk writing occures in the beginning of the move operation which in my case for about 10k e-mails took less than 10 min but after this step no disk activity, only CPU at 100% and memory usage growing ... i guess it has to do with the internal email messages list or somtething like that...and this also is valid for both windows and linux 

PS. I'm not using addons/extensions because the base e-mail client has all i need 

PS2. My PC has a 1.8 GHz Intel CPU, 512 MB RAM and a SATA hdd so it should do fine with these resources running most office applications ... this is to eliminate hardware out of the equation

Thanks and i will get back with the results to test Q7.

Comment 7

[Bozhan Boiadzhiev]

same situation here.
huge amount of emails from arpwatch, caused bye some virus produced about 15000 emails in my inbox. Deleting (moving to trash) takes about 15min.
Until i get my emails from mail server is very hard to do something else with thunderbird.
Deleting emails from Trash takes about same time!
CPU at 100%, and Thunderbird even hangs.
Intel Celeron 2.4GHz, 512MB RAM.
Dont use and extension.

Comment 8

[Wayne Mery (:wsmwk)]

shouldn't be sev=enhancement

dupe of bug 296453?

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Bug 296453 is about IMAP while we have a POP3 issue here. No idea if the cause is the same for both protocols.

Comment 10

[WADA:World Anti-bad-Duping Agency]

Bug summary of Bug 296453 still has "IMAP", but it's not IMAP unique issue.
See Bug 429753 & Bug 393123(both were closed as DUP of Bug 296453).

I could observe "Shift+Delete takes long time & CPU 100% while Shift+delete & UI lock while Shift+delete" at last, with mails of long thread(all mails has same subject). See Bug 452221. See also Bug 429753 for "long thread".
In test of Bug 452221, I couldn't observe increase of virtual memory size/huge virtual memory size. It is possibly due to "Shift+Delete" case or "single thread" case or "no In-Reply-To: header" case.

To bug opener & all problem reporters:
"Many threads of very long thread" exist in your mail folder? 

Comment 11

[Wayne Mery (:wsmwk)]

I'm tempted to dup because I think they are the same, but being conservative - sticking with dependency and confirming to get this off UNCO list.  Please reevaluate after bug 296453 has a patch.