when many moving messages from one folder to another cpu is at 100% and process taking huge amounts of time

RESOLVED DUPLICATE of bug 296453

Status

Thunderbird
Message Compose Window
--
critical
RESOLVED DUPLICATE of bug 296453
11 years ago
6 years ago

People

(Reporter: Seceleanu Cristian, Unassigned)

Tracking

({hang, perf})

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [waiting on bug 296453])

(Reporter)

Description

11 years ago
User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4
Build Identifier: version 2.0.0.0 (20070326)

Hi, i'm using Linux and thunderbird as pop3 client.
I'm a system admin and daily i get over 3k e-mail messages in my inbox and from time to time i need to move them to other folders (as when they accumulte intro one folder it works much much slower). 
The problems is that when i move say 10000 messages from a folther with 100000 messages it take some 2h to complete the operation and it eats over 300MB RAM to do it and of course cpu is at 100% usage.
This is an optimization issue, not a bug, and i'm sure you can fix it provided you are interested in improving this excellent e-mail client.

Reproducible: Always

Steps to Reproduce:
1.Fill a folder with more than 100k messages
2.Create another folder or subfolder, it does not matter
3.Move 10000 messages from the 100k messages folder to the newly created one

Actual Results:  
It took over 2h time to complete, during which it used 100% CPU and 300MB RAM

Expected Results:  
Finish in less than 10 min with less resources
When you are getting such an amount of messages day by day have you ever compacted your folders? It's only a vague feeling but if you have never did that please try it now: http://kb.mozillazine.org/Compacting_folders

Does it solve your issue? If not, where is your destination folder located? Locally? Another account? IMAP account?
(Reporter)

Comment 2

11 years ago
I compact my folders some 5 - 10 times a day as it also takes some 5 minutes each time and i don't want to do it too many times if you understand what i mean....

I have 5 accounts, 3 have separate folders, 2 use the local folders, i'm experiencing this with all of them (i have one with 300k email messages, another with some 180k and a couple more folders with 20k )

Deleting the older messages is not an option for me as i need to keep them for many reasons. Don't get me wrong, i tried using outlook :) and it dies when i hit the 32k limit in one folder :) so it's far better using thunderbird but still i think that it can be improved regarding this issue 

PS. All 5 are pop3 accounts.
(In reply to comment #0)
> it take some 2h to complete the operation and 
> it eats over 300MB RAM

WORKSFORME with Tb 2.0.0.x/Seamonkey 1.1.x/many Tb trunks builds on my Win-XP SP2 PC.

To Seceleanu Cristian(bug opener):

(Q1) Which do you mean by "300MB RAM", real memory used by Tb, or virtual memory used by Tb?
(Q2) When "real memory used by Tb", how many physical memory do you have?
Was heavy swapping observed while mail moving?
(Q3) When "virtual memory used by Tb", do you mean that very large VM size increase was observed after start of move mails/while moving mails?.
(Q4) Does your problem occur even with newly created profile?
(Q5) Do you use aniti-virus software? If yes, will "excluding of mail folder files from scan target" alter phenomenon? Try next.
 1. Say source folder="AA" and target folder="BB"
 2. Esxclude "AA", "AA.msf", "BB" and "BB.msf" from scan target
 3. Move many mails from AA to BB
(Q5) Do you use "Google Desktop"?
(Reporter)

Comment 4

11 years ago
Hi,

A1 to Q1:
300MB real RAM
A2 to Q2:
512 MB RAM
Heavy swapping up to 800 MB swap sometimes (i tried once to move 30k messages in one move ) and i notice something else...on normal usage, not moving e-mails, just opening folders, reading and receiving e-mail i notice some 250 - 300 M swap usage  and about 180 - 200M RAM usage (after i close thunderbird i notice more free RAM with about 200 M and swap returns to 0  after being 250 - 300 MB used)
A3 to Q3:
yes
A4 to Q4:
yes, after a couple of days of e-mail downloading when i get all e-mail in my inbox it does the same thing
A5 to Q5:
No antivrirus, i am using linux, there is no need for that here and no i do not use google desktop

Also i would like to mention that in total my thunderbird profile uses 4.6 Gbytes of disk space (99.9% of disk space are most likely my e-mail messages) -> this is probably the cause for A2 above but still even in my case i think that it could be optimized to eat less resources... 

I'd like to mention that i am willing to help fix this bug either by writing code or providing some ideas to thunderbird developers so that the fix can be included in trunk asap.

Thanks
(In reply to comment #4)
> A4 to Q4:
> yes, after a couple of days of e-mail downloading when i get all e-mail in my
> inbox it does the same thing

My "new profile" implies "no extension/add-on", and it doesn't mean "newly created but customized".
(Q7) Will your problem be re-produced by following test?
1. Create new Thunderbird profile.
2. Define a POP3 account(dummy server, No Global Inbox).
3. Terminate Thunderbird
4. Copy file of "Inbox"(not "Inbox.msf") in Mail directory, which is you are
   using daily, to Mail directory of the newly created profile.
5. Restart Thunderbird, and click "Inbox", and confirm "Inbox" is healthy.
6. Define a mail folder (say "Test")
7. Move many mails from "Inbox" to "Test"
(Reporter)

Comment 6

11 years ago
Hi,

I will perform this test but unfortunately i have to leave town for about 2 weeks. I will let you know how this test turned out when i get back.

Also i'd like to mention that i have done something similar to Q7 before switching entirely to linux, i backed up my profile from windows then after switching to linux i used thunderbird --profilemanager and pointed to the folders from windows ... it worked ok, i mean folders were healthy, i could see all e-mail messages, i compacted the folders just in case, but when moving e-mail messages from one folder to another the same problem occured.

...i think i forgot to mention that this happened on both windows and linux versions of thunderbird

Another thing i noticed is: actual disk writing occures in the beginning of the move operation which in my case for about 10k e-mails took less than 10 min but after this step no disk activity, only CPU at 100% and memory usage growing ... i guess it has to do with the internal email messages list or somtething like that...and this also is valid for both windows and linux 

PS. I'm not using addons/extensions because the base e-mail client has all i need 

PS2. My PC has a 1.8 GHz Intel CPU, 512 MB RAM and a SATA hdd so it should do fine with these resources running most office applications ... this is to eliminate hardware out of the equation

Thanks and i will get back with the results to test Q7.

Comment 7

11 years ago
same situation here.
huge amount of emails from arpwatch, caused bye some virus produced about 15000 emails in my inbox. Deleting (moving to trash) takes about 15min.
Until i get my emails from mail server is very hard to do something else with thunderbird.
Deleting emails from Trash takes about same time!
CPU at 100%, and Thunderbird even hangs.
Intel Celeron 2.4GHz, 512MB RAM.
Dont use and extension.

Comment 8

10 years ago
shouldn't be sev=enhancement

dupe of bug 296453?
Severity: enhancement → major
Depends on: 296453
Bug 296453 is about IMAP while we have a POP3 issue here. No idea if the cause is the same for both protocols.
Keywords: hang
Bug summary of Bug 296453 still has "IMAP", but it's not IMAP unique issue.
See Bug 429753 & Bug 393123(both were closed as DUP of Bug 296453).

I could observe "Shift+Delete takes long time & CPU 100% while Shift+delete & UI lock while Shift+delete" at last, with mails of long thread(all mails has same subject). See Bug 452221. See also Bug 429753 for "long thread".
In test of Bug 452221, I couldn't observe increase of virtual memory size/huge virtual memory size. It is possibly due to "Shift+Delete" case or "single thread" case or "no In-Reply-To: header" case.

To bug opener & all problem reporters:
"Many threads of very long thread" exist in your mail folder? 

Comment 11

10 years ago
I'm tempted to dup because I think they are the same, but being conservative - sticking with dependency and confirming to get this off UNCO list.  Please reevaluate after bug 296453 has a patch.
Status: UNCONFIRMED → NEW
Component: General → Message Compose Window
Ever confirmed: true
Keywords: perf
QA Contact: general → message-compose
Whiteboard: [waiting on bug 296453]

Updated

9 years ago
Severity: major → critical

Updated

6 years ago
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → DUPLICATE
Summary: when moving messages from one folder to another cpu is at 100% and process taking huge amounts of time → when many moving messages from one folder to another cpu is at 100% and process taking huge amounts of time
Duplicate of bug: 296453
You need to log in before you can comment on or make changes to this bug.