336954 - "Compacting folder" with 4000+ messages from IMAP account drives CPU 100%

Status: RESOLVED WORKSFORME

(Thunderbird :: Mail Window Front End, defect)

Description

[Anthony Howe]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3

Thunderbird version 1.5.0.2 (20060308)

I have an IMAP account that collects a large quantity of email. Every two or three days I have to treat and empty this box containing 4000 or more messages. I have "mark it as deleted" set. The first step to "delete" the messages works fine, but when I proceed to use File > Compact Folders to purge those marked messages, the CPU goes 98% for a long long time (15 minutes or more, I usual go do the ironing at this point) but it will eventually complete.

On the IMAP server side, I've confirmed that the 4000+ messages are quickly deleted shortly after issing the command, so it would appear that Thunderbird spends all its time in some local post processing of internal structures.

I've consistently been able to duplicate this for several months and only now have decided to report it, since it didn't seem to be a version specific problem.

Reproducible: Always

Comment 1

[David :Bienvenu]

do you have that folder configured for offline use? If so, we'r probably compacting the offline store - that should never take 15 minutes, but it is probably involved. The other thing we do when compacting the offline store is repair it if it has become damaged by re-downloading messages, if you're online.

Comment 2

[Anthony Howe]

No. The folder is NOT configured for offline use.

This comment doesn't make sense or reads poorly:

"The other thing we do when compacting the offline store is
repair it if it has become damaged by re-downloading messages, if you're
online."

Does this concern an online folder? Or do you mean when online, the offline store is resynced? Regardless, that would be horribly slow and wasteful when your deleting messages a lot of messages.

Anyways the folder is used in an "online only" capacity and there is no network activity that can be seen.

Comment 3

[David :Bienvenu]

Not sure what doesn't make sense about that last statement. The offline store refers to the file where we store offline copies of messages in online (IMAP) folders, if you've configured TB to do so. Damaged means the offline store has somehow become corrupted, so that messages TB *and* the user thinks they have offline are not in the offline store. This should be extremely rare, but it's a situation that needs to be repaired, and it's a situation we detect while compacting an offline store. If you don't have an offline store, then that's not what's happening to you.

When you compact using the imap delete model (such that you see deleted messages in the UI), we do end up having to delete 4000 messages from the DB, and adjust the view, etc. It shouldn't take 15 minutes, of course. What happens if you delete the 4000 messages, then load a different folder, and then right click on the folder you deleted the 4000 messages from, and pick compact this folder? If that's relatively fast, then we could blame updating the UI for 4000 deletes...

Comment 4

[Anthony Howe]

I've had to wait until I've had enough email in this folder to try as you suggested. Now I...

a) Select the Inbox folder, I select & delete 3806 messages, which get marked quickly enough.

b) I select a different Inbox of another account.

c) Now I right click on the folder with 3806 marked messages and select "Compact this folder".

The CPU spikes and the egg timer appears and I can't do anything with Thunderbird until it finishes. It took 16 minutes before I regained control of Thunderbird and the CPU returned to normal. I could use other applications, like Firefox, ssh, text editor, etc. while it worked.

Comment 5

[Motin / Fredrik]

bug 320515 comment 5 could be of help

Comment 6

[Anthony Howe]

All my folders are IMAP folders in my unix account space, not local to my Windows hard disk.

Today I had enough mail one Newsletter collection folder with 4461 messages (15MB) to test version 2.0.0.0 (20070326). Things were much better.

a) The unread mail counter still takes a long time to count down to 0, why it can't just set itself to zero (or whatever the new value is) without doing some sort of repeative screen update I don't know. This takes about 40 seconds or so.

b) The actual "purge" (compact folder operation) took less than 2 minutes. I did switch from Thunderbird to some other app. and then tried to come back to Thunderbird. Thunderbird would not redraw the blank screen until AFTER the purge was complete ("not responding " appears in the process list), but it didn't take as long as it once did.

I have one more large Newletter folder (4699 messages 23MB) I'm going to purge and see if the observations remain consistent.

Comment 7

[Anthony Howe]

Following up to my previous post concerning another _large_ IMAP folder:

a) Time to mark 4699 messages deleted (red cross & strikeout) instant; time to mark 4699 messages read (from unread) 75 seconds on the stop watch.

b) Time to purge 4699 messages 50 seconds on the stop watch (not as bad as I thought). Switch from/to process though was non-responsive while during this time. 

Things have certainly improved in this area. I think the issues that remain are more cosmetic in nature, though as the size of mail folders increase, these times will get longer. I have some special collection folders I could make copies of and test that have 85000+ messages.

Comment 8

[Wayne Mery (:wsmwk)]

(In reply to comment #7)
> Following up to my previous post concerning another _large_ IMAP folder:
>...
> Things have certainly improved in this area. I think the issues that remain are
> more cosmetic in nature, 

=> WFM per reporter