217299 - "Delete Mail marked as Junk in Folder" very slow with large number of messages

Status: RESOLVED FIXED

(MailNews Core :: Database, defect)

Description

[Ulf Dunkel]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5b) Gecko/20030822
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5b) Gecko/20030822

I use to get about 2500 junk mails per day in one of my accounts (no joke).

When I chose the acc. menu entry from the Tools menu, the mouse changes to an
hour glass and Mozilla crashes. I give it about half an hour and check the
Windows task manager - it is dead and gone. When I click the closer in the upper
right corner of the Mail windows, not only Mail is closed but the whole Mozilla
suite is exited. (I use Quick start, of course.)

Reproducible: Always

Steps to Reproduce:
1. Collect MANY junk mails - this bug shows no effects for a handful of junk
mails only.
2. Chose "Delete Mail marked as Junk in Folder".
3. Byebye ...

Actual Results:  
Mozilla hangs up and has to be terminated somehow.

Expected Results:  
I expected Mozilla to move the about 2500 Junk mails to the Trash folder.

Comment 1

[David :Bienvenu]

Are you using an IMAP or POP3 mail server?

I'm a little confused - you say that you crash when you pick the acc ("ount
settings"?) menu item from the tools menu (which implies you're using
Thunderbird). But then you say that you crash when you choose "Delete Mail
marked as Junk in folder". The first thing sounds like a hang, and sounds unrelated.

Comment 2

[Ulf Dunkel]

Hi. I use POP3 servers for my accounts.

Sorry for my maybe misleading descriptions - I am not a native English speaker.

I use Mozilla, not Thunderbird. In the Mozilla "Mail & Newsgroups", I select the
acc(ording) menu item "Delete Mail marked as Junk in Folder" from the "Tools" menu.

Does this help?

Comment 3

[David :Bienvenu]

Oh, yes, that helps a lot, thx. It sounds a bit like the deleting of the 2500
junk messages might just be taking a really long time, and that we crash when
you try to close Mozilla while the deleting is going on.  How big is your INBOX
file on disk? Have you tried compacting it?

Comment 4

[Ulf Dunkel]

At the moment, the values for the according mail account are as follows:

Inbox      94.045 KB, showing 56 messages, 56 unread
Inbox.msf      86 KB
Junk       35.663 KB, showing 5205 messages, 5180 unread
Junk.msf    1.807 KB

I select the Junk folder and chose "Delete Mail marked as Junk in Folder" then.
The menu entry gets selected (inverted), the mouse form stays as an arrow, the
Tools menu box is not redrawn or closed and Mozilla hangs.

When I want to close it, Mozilla crashes completely.

2. step:
Now I compact the Inbox folder (works very fast) as well as the Junk folder
(needs about one minute on my 2.4 GHz machine). The new values are:

Inbox         832 KB, showing 57 messages, 57 unread
Inbox.msf      19 KB
Junk       35.351 KB, showing 5207 messages, 5182 unread
Junk.msf    1.727 KB

Then I retry deleting the Junk mails in the Junk folder as described above. The
same behaviour. Taking a look at the task manager, I see that Mozilla takes 99%
of CPU power. After 2 minutes I kill Mozilla as it cannot be accepted that a
"rather simple" delete procedure occupies the whole computer for this period of
time.

Conclusion: I wonder if the algorithm beneath this Tools function has to be
reviewed. If I can be of any further help, please let me know. I have NO
PROBLEMS in receiving a real big bunch of Junk mails every day. ;-)

Comment 5

[David :Bienvenu]

Ulf, thx for the info. If I understand you correctly, this operation will move
about 5200 messages from the junk folder to the trash folder (assuming almost
all messages in the junk folder are junk). How big is your trash folder? Have
you emptied it recently? I can imagine if your trash folder is large, then
moving 5200 messages (35MB) could take some time. I suspect you're right that
"Delete Mail marked as Junk in Folder" code needs to be looked at, and I'll do
that now.

Comment 6

[David :Bienvenu]

OK, the code that does this is written in js, and is definitely not efficient -
it works by iterating over the messages and adding the junk ones to the
selection, and then deleting the selection. Adding things to the selection can
be very very slow and writing all this code in js is slow too. I still wouldn't
expect it to be this slow. What kind of CPU do you have?

Comment 7

[David :Bienvenu]

heh, never mind - when I said "adding things to the selection can be very very
slow", that was an understatement. It's incredibly slow - we fire a selection
changed event for every message added, and that takes a really long time. I
think I'll try rewriting this code to just accumulate the msg keys and call
delete on the folder directly, or try to suppress the selection changed events
somehow.

Comment 8

[David :Bienvenu]

Attachment: proposed fix

this patch makes it so we disable the on select notifications while building up
the selection. This speeds this up immensely, so that we get to the delete part
pretty quickly. Once the delete is finished, we spend an inordinate amount of
time cleaning up the selection, but it's a lot better than it was before.

Comment 9

[(not reading, please use [e-mail] instead)]

Comment on attachment 130475 [details] [diff] [review]
proposed fix

r/a=sspitzer for 1.5 beta (or 1.5 final)

Comment 10

[David :Bienvenu]

fix checked in - please try tomorrow's build. It should be a lot faster to get
to the part where it deletes all the messages. There will be a significant delay
after it finishes deleting the messages - it might be as long as a minute or
more, I'm not sure because I haven't tried it on a release build. But overall,
the whole process should be a lot faster.

Comment 11

[Ulf Dunkel]

My according trash file is 8 KB, trash.msf has 8 KB, too. :-)

Comment 12

[Ulf Dunkel]

Hi David.

Sorry for my delay in answering your questions. My CPU has 2.4 GHz.
I will try todays build as soon as it is available.

If you could really fix or enhance this feature, take my biggest THANK YOU for
this birthday present for me (27.08.). :-)

Comment 13

[Matthias Versen [:Matti]]

Attachment: screenshot

Bienvenu: Is it possible that this patch caused this error message that I got
today (and never before) ?

Comment 14

[David :Bienvenu]

Matti, I've gotten that message w/o this change (usually when auto-compact is
running and get new mail decides to filter into a folder that's being
compacted). This change really has no effect on the way messages are deleted,
just what happens to the selection ui before they're deleted. If you ran this
command on the spam folder at the same time as check for new mail fired and
tried to move a message to the spam folder, you'd get that error message,
independent of this change.

Comment 15

[Matthias Versen [:Matti]]

Auto-Compacting is disabled (because of the very frequent error messages)

I started Mozilla today and it downloads all Mails on startup and got this error
message while Mozilla downloaded the mails.

I will see if I get that tommorrow morning again.
(because my Accounts select many incoming SPAM messages over night)

Comment 16

[David :Bienvenu]

if you didn't execute the "Delete Mail marked as Junk in folder" command, than
this change could not possibly be the cause of the error message.

Comment 17

[Matthias Versen [:Matti]]

I have the "Automatically delete messages marked as junk" set to 14 days but i
clean my spam folder every day (manually delete). It could be caused by a
message that I received today with a wrong date (2002 instead of 2003)

Comment 18

[David :Bienvenu]

That would explain it - it would still be unrelated to this change.  I'm not
sure when the auto-purge stuff runs - it would be unfortunate if it ran at the
same time as get new mail was running, like auto-compact, i.e., the time it's
most likely to cause conflicts :-(

Comment 19

[Ulf Dunkel]

Now I tried Build 2003082804, as promised.

D:\MyData\Mozilla\Profiles\UlfDunkel\5byc6837.slt\Mail\dsd.net:
Inbox       16.372 KB  Unread:    0, Total:    0
Inbox.msf       28 KB
Junk        51.722 KB  Unread: 7497, Total: 7530 (1 of which is NOJUNK!)
Junk.msf     2.502 KB

No reaction. Then I opened the Task Manager and could see that Mozilla tried and
got about 75& of CPU time as another process was busy in the same time (getting
25%). No redraw of the Tools menu, no change of the mouse form.

After approx. 10 seconds, the Mozilla Mail&Newsgroup window was refreshed and I
now listen to the hard drive moving files (or rather shuffling the Junk stuff
from the Junk file to the Trash file).

I let Mozilla proceed but there is no further process as I can see by reviewing
the data folder. Now the Junk and Trash files have not been updated for about 10
minutes already and I terminate Mozilla.

2. trial:
Oh, what's that?!? The Junk folder builds a new summary and shows only 15 Junk
mails (which have arrived in the meantime). The Trash folder creates a new
summary and shows more than 7500 messages - as expected.

Because the Junk file was still very large, I deleted the Junk.msf and restarted
Mozilla. Now that I selected the Junk folder, Mozilla told me "Building summary
for Junk" - for about 10 minutes. Then I went writing this report. ;-)

My conclusion for this time:
Your patch helped "a little bit" but in fact there must be another timewasting
whole in the sourcecode. After the Junk mails have been swapped to the Trash
file, I would have expected the Junk file to be compressed automatically. In
fact, Mozilla hung for more than 15 minutes after the hard drive activities had
ended.

Comment 20

[David :Bienvenu]

yes, I knew there was a delay after the delete happened:

>There will be a significant delay after it finishes deleting the messages - it
>might be as long as a minute or more, I'm not sure because I haven't tried it
on >a release build. 

For me, on my 1.6 Mhz machine, with 2500 messages on a debug build, it was not
more than 2 minutes, so I'm a little surprised that it was more than 15 minutes
for you, but the delay in that selection code might be non-linear. But I think
there's some improvement because what took more than half an hour before is only
taking 10 seconds (getting to the part where we actually delete the messages).
I'm not quite sure how to fix that last part since it's all happening in the
tree selection code...

Comment 21

[Ulf Dunkel]

I wonder if you can keep control over the mouse form - but wouldn't it be a good
idea to let the mouse show the busy state until this tree stuff is done?

Comment 22

[David :Bienvenu]

Attachment: speed up post delete processing

this patch speeds up the code after the delete happens by clearing the
selection after we issue the delete.

Comment 23

[David :Bienvenu]

changing summary - re the stopwatch, because the actual delete is async, it
makes it harder to turn off the stopwatch, so we'd have do this for the general
case of delete/move/copy, not just this particular case.

Comment 24

[(not reading, please use [e-mail] instead)]

Comment on attachment 130552 [details] [diff] [review]
speed up post delete processing

r/a=sspitzer

Comment 25

[David :Bienvenu]

fixed

Comment 26

[Scott MacGregor]

FYI: I've taken this last patch on the Thunderbird 0.2 branch (the first patch
was already on the branch)

Comment 27

[[e-mail]]

Might it be slightly more efficient to unsuppress select events after clearing
the selection?

Comment 28

[Ulf Dunkel]

Hi David, it's me again.

I wanted to test the fixed functions on August 31st build, but the
Win-Installer.exe was damaged. So I tried the build 2003090104 today.

Here is my test and result:

Inbox      6641 KB = 0/0 messages
Inbox.msf    55 KB

Junk      35164 KB = 5963/5980 messages (all junk)
Junk.msf   2102 KB

Trash         1 KB = 1/1 message
Trash.msf     2 KB

I selected the Junk folder of my dsd.net account and clicked on the "Tools" menu
entry "Delete Mail Marked as Junk in Folder" then. For about 10 seconds, the
menu box was not redrawn nor changed the mouse form. After that, the Mail &
Workgroups window was refreshed and - surprise - the selected folder no longer
was "Junk" of my dsd.net mail account, but was "Junk" in my first mail account
(calamus.net).

I listened to the hard drive which worked for about 90 seconds, then the dsd.net
Junk folder was EMPTY and the according Trash folder was filled with Junk mails.

Congratulations!

Another test with the Junk folder in my calamus.net account (about 500 Junk
mails) showed an acceptable duration time of about 3 seconds. :-)

I would be glad if there was any chance to enhance the speed of message database
actions even more. But for the moment, I am glad that you managed to speed up
thinks like this.

Comment 29

[David :Bienvenu]

*** Bug 217622 has been marked as a duplicate of this bug. ***