Closed Bug 429753 Opened 16 years ago Closed 16 years ago

Thunderbird is slow to delete multiple emails, (IMAP deleting, deletion)

Categories

(Thunderbird :: General, defect)

Product:
Thunderbird
Component:
General
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 296453

People

(Reporter: justdave, Assigned: bugmil.ebirol)

Details

(Keywords: perf) 

+++ This bug was initially created as a clone of Bug #393123 +++

Thunderbird is extremely slow when deleting lots of emails, hundreds or thousands of them.  I am using IMAP

The particular task I was trying to accomplish was deleting a few thousand emails (from a folder into the trash, although it also seems slow deleting selectively those emails from the trash also).  

CPU seems to spike to 100% and stay pegged while emails are being deleted.  Thunderbird tends to hang until the deletions have completed.  

I've had problems doing this for various Thunderbird releases for the last year or so, not just the current release.

Reproducible: Always

Steps to Reproduce:
1. Open a mailbox with several thousand messages in them
2. Select a thousand or two of them
3. Hit the Delete key
Flags: blocking-thunderbird3?
There's a Shark dump at
http://people.mozilla.com/~justdave/Time_Profile_of_thunderbird-bin_200804101056.mshark

(too big to attach here [1.6MB, compressed])

Please explain detail of "slow" or time related phenomenon or "CPU 100%".

As I wrote in Bug 393123 Comment #6, delete is roughly; 
 (B) IMAP (care on "move to a folder on same non-Gmail IMAP server" only)
   1. Request "copy" to IMAP server for all mail to be moved
      File I/O(physical file read/write==copy) occurs at IMAP server
      IMAP server has to update "index data" for move target folder
   2. Re-sync and refresh of ".msf"(mail db) for move target(Trash)
   3. Request "flag \Deleted" to IMAP server for all mail which is moved
      IMAP server has to update "index data" for source folder
   4. Re-sync and refresh of ".msf"(mail db) for source folder
   (When Shift+Delete, step 1 & step 2 are bypassed) 
And you say next in Bug 393123 Comment #8.
> at the point when I hang, all traffic to the IMAP server seems to have stopped.
> Process Sampling indicates Thunderbird is stuck in nsMsgDatabase::ReparentChildrenOf when it hangs.

(Q1) Hang? Or "CPU 100% while long time of delete"(delete completes)?
(Q2) Slow at which step? "CPU 100%" at which step? How long your PC stays "CPU 100%"? As you say nsMsgDatabase::ReparentChildrenOf, I guess step 4.
 (I think not "display as thread". "mail db data update for threading" instead.)
 (So it probably happens even when you are not in "Thread display mode".       )
(Q3) How big is your IMAP folder/.msf file?
     As written in Bug 347665 and Bug 384360, re-sync in IMAP takes long.
     And as written in Bug 226730 Comment #6, mail db close/reopen(=re-sync)
     seems to be executed for each delete of 300 mails.
     So, even if same issue as Bug 393123 on POP3 folder, situation can become
     very worse when delete of many mails in huge IMAP folder.

> at the point when I hang, all traffic to the IMAP server seems to have stopped.

"Blocked-UI while other task such as re-sync of IMAP folder" is already reported phenomenon in other bugs such as Bug 347665, Bug 384360, and bug summary of Bug 384360 refers to it. This is mainly task scheduling issue and different/independent issue from this bug, I believe. Sorry but I don't know whether bug to resolve the blocked-UI issue is already opened or not.

"traffic to the IMAP" seems to be stopped like bocked-UI while some tasks.
This may interfere frequent re-sync of IMAP folder due to delete of many mails.
Dave Miller, problem is easily re-produced? How many mails are required to re-produce? I can test with Gmail IMAP account, and I can get IMAP protocol log with timestamp by DebugView(I use MS Win) to see how frequent re-sync is executed and how long a re-sync takes while delete of many mails.
Yes, easy to reproduce.  The situation where I run into it is that I get mail from all of the cron jobs on all of the servers at Mozilla, which usually amounts to a few thousand pieces per day, which all filters into the same mailbox.

Typical mode of operation is I'll throw something in the search box to match a mail that I see so it filters it so I see only the reports from that specific job, check a couple of them at random to see what it's complaining about, and deal with the problem if necessary, then I do a select-all and hit the delete key. I usually do a delete instead of shift-delete because the re-sync after a shift-delete almost always failed (it would disappear until you restarted thunderbird then they all came back because the server-side mailbox purge never happened), but that's another bug, and for all I know it might be fixed now, I'll have to test it (been ages since I bothered).

So I have a mailbox with 20K messages in it, and I'm usually deleting a few thousand at a time.

(In reply to comment #2)

> (Q1) Hang? Or "CPU 100% while long time of delete"(delete completes)?

CPU at 100% and the Thunderbird app is complete unresponsive with the spinning beach ball cursor for the duration of the delete to complete (which takes forever and a day, see below)

> (Q2) Slow at which step? "CPU 100%" at which step? How long your PC stays "CPU
> 100%"?

I don't know which step, I'm not a Thunderbird developer. :) All of the dumps/traces I've provided so far have been at the direction of folks on #maildev telling me what to do to produce them (other than the tcpdump because I'm a sysadmin so I know how to do that). :) But if I delete around 1000 messages, it's stuck at 100% CPU for around a half hour usually.  I've had it go 90 to 120 minutes before doing 6000 to 8000 messages.

> (Q3) How big is your IMAP folder/.msf file?

-rw-r--r--@ 1 dave  dave    91M Apr 18 21:46 Cron Job Output
-rw-r--r--  1 dave  dave   9.1M Apr 18 21:56 Cron Job Output.msf

Right now there's 18,585 messages in it.

> > at the point when I hang, all traffic to the IMAP server seems to have stopped.
> "traffic to the IMAP" seems to be stopped like bocked-UI while some tasks.
> This may interfere frequent re-sync of IMAP folder due to delete of many mails.

No, this is based on a tcpdump of traffic between me and the IMAP server.  There is no traffic in either direction for the entire duration of time that the CPU is at 100%. Which makes me think whatever's causing the time to be spent is happening locally.

Thanks for detailed explanation of situation and phenomenon.

nsMsgDatabase::ReparentChildrenOf is defined in nsMsgThread.cpp.
> http://lxr.mozilla.org/seamonkey/source/mailnews/db/msgdb/src/nsMsgThread.cpp#584
If root parent of a big thread is deleted, I guess ReparentChildrenOf will take longer because all children have to be updated. And it's a CPU bound task. It may be a reason of "CPU 100%" phenomenon.

> The situation where I run into it is that I get mail from all of the cron jobs
> on all of the servers at Mozilla, which usually amounts to a few thousand
> pieces per day, which all filters into the same mailbox.
>(snip)
> So I have a mailbox with 20K messages in it, and I'm usually deleting a few thousand at a time.

Does it mean that there are many many threads in a mbox and many many big threads and you delete parent mail & some children from many threads? If so, I think it can explain why I could't observe phenomenon by my test. I tested with spam mails, which never has big thread. I'll try to generate test mails of many big threads first.

Can you test with local copy of IMAP folder(same case as Bug 393123)? Can you observe phenomenon with local copy of the IMAP folder you experienced problem?

(Off-Topic)
> re-sync after a shift-delete almost always failed
> (it would disappear until you restarted thunderbird then they all came back
>  because the server-side mailbox purge never happened), but that's another bug
IFAIR, I saw bugs for similar issues recently. They are probably still open.
 1. Expunge is not issued when Shift+Delete.
    So no difference from "Delete" when "Mark it as deleted" model.
 2. Locally deleted IMAP mail(while offline mode) comes back, because re-sync
    is executed before request of "flag \Delete" based on local delete.

Brief report of quick test with long thread.
 - Tb trunk 4/20 build, MS Win-XP
 - Local mail folder.
   All mails have same Subject:, so only one very long thread in a mbox.
   No In-Reply-To: header, so thread depth=1.　
 - Time to display thread view. "CPU 100%" continued while threading.
   (1)  4000 mails :  20 sec.                     2.2 MB, msf size=1 MB
   (2)  8000 mails :  70 sec. (1 minutes 10 sec)  4.3 MB, msf size=2 MB
   (3) 12000 mails : 410 sec. (6 minutes 50 sec)  6.5 MB, msf size=3 MB
   Not linear. O(exp(n))?
   Threading seems to consume CPU power very much.
 - After completion of threading, delete of root mail finished at a glance.
 - Re-threading after unthread took same long time.

> I get mail from all of the cron jobs

Many mails have same subject? (very very long thread exists?)
How long does threading of the IMAP folder take?
Does search folder which refers to the IMAP folder exist?
(In reply to comment #6)
> > I get mail from all of the cron jobs
> 
> Many mails have same subject? (very very long thread exists?)

Yes, there will be lots of messages with the same subject, in fact, usually when I'm deleting a few thousand at once, everything I'm deleting will have the same subject (or two or three similar subjects).

> How long does threading of the IMAP folder take?

I don't use threading on that mailbox.  Does TB do it anyway even though I haven't asked for it?  How would I tell how much time that's taking?

> Does search folder which refers to the IMAP folder exist?

No, no search folders set up on that account.
Assignee: nobody → ebirol
Bug 296453 seems very similar to this.
Bah, looks like a dupe.  And I even touched it, too.  Wonder how come I couldn't find that when I was looking around before filing this one?
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → DUPLICATE
(Off-Topic)
To Dave Miller (MoCo):  
Andrew Sutherland has took bug 296453, based on your detailed report, and by your "duping to bug 296453". But I prefer "duping bug 296453 & Bug 393123 to your bug", because you are the first person who referred to very very long thread and made it clear that client side only issue even when IMAP folder(mail DB=.msf update issue when very long thread).

FYI.

"Slowness in threading of very huge thread" itself was being handled by Bug 226730.
The (severe) slowness can easily be observed at least following stages;
(1) Rebuild Index of many mails of same Subject:(including preceding "Re:" case)
(2) Rebuild Index of very huge/deep thread with "In-Reply-To:" headers
(3) View a folder of (1) or (2) with threading (Bug 226730 is for this issue)

(1) and (2) relates to mail DB(.msf file content) construction. So (1) and (2) relates to this bug's issue, although (3) doesn't have relation to this bug's issue. 
I've made the same constatation (TB2 and 3 latest version) with pop3 account

(i get lot of "undelivered mail" because i manage different mailing lists)

when I want to erase a lot of mail (last time 10K ) with no thread, TB start working (turning on the top right) after i get a white windows and it takes hours before I get a working windows (it's using half of the CPU because I get a pentium3 )

it's the same if I try to compact folders

(may be the 8.5 Go thunderbird data is a start of the answer ? ;-)))

suggestion :

make the erase task back ground ?

possibility to add a configurable trigger to automatiquely erase mail (older than...) to the folder properties (useful for the undelivered mail to keep an eye on the mailing list state but not be bothered with the uge amount of mail to delete manualy) and for the junk folder and the trash folder

possibility to add a trigger to the trash folder to compact at startup or closing TB

PS may be it's already done but as standard use I don't know where to active or get the plug in to do that
 
new try:

deleting 6279 mail fron a 10K folder

moving xxx of 6279 messages start nicely without massive use of CPU

when moving 6041 of 6279 messages, the cpu use climb to 50% and TB lock no more response

look like the re-indexation of the base is blocking TB

this time is took 5 mn to have the job done
Flags: blocking-thunderbird3?
I'm having the same problem as described in the initial report in the latest TB 45.1.0. When trying to delete ~60 000 messages, TB gets unresponsible for some time and then shows "stop scrip" popup.

These messages are the reporst of some monitoring robots, so most of the messages have same subject (threading issue mentioned up above, but I don't use threaded view for this folder).

It seems that deleting oldest mssages first in a small (~4000) batches allows for smaller hangups and eventually deletion of messages, but it's a pain.

I see that the issue is resolved via a chain ending with bug538378, but I still face this problem.
You need to log in before you can comment on or make changes to this bug.