693659 - Main window completely locks up when trying to delete thousands of messages

Status: NEW

(MailNews Core :: Backend, defect)

Description

[David Tenser [:djst]]

In my mozilla.com IMAP folders, I have one folder with (sometimes) over 15000 email messages. (These are nagios error emails that I rarely look at.)

When selecting all of those messages and hitting Delete, the UI completely hangs for over a minute. This doesn't seem like the ideal way of handling this. Maybe put the actual work in a separate worker thread and show a progress bar instead?

Comment 1

[David :Bienvenu]

this is a known bug, with several causes, and I've been looking into it recently. Two main issues are that some of the change notifications between c++ and js aren't batched, and there should be one undo action for this, instead of N where N is the number of messages.

Shift delete is probably quite a bit faster, if you don't care about the trash.

Comment 2

[David Tenser [:djst]]

Thanks for the workaround tip; I'll try shift-delete next time. Oh, and sorry for filing a (I'm guessing?) dupe!

Comment 3

[Ludovic Hirlimann [:Usul]]

(In reply to David :Bienvenu from comment #1)
> this is a known bug, with several causes, and I've been looking into it

David what's the master bug for this ?

Comment 4

[David :Bienvenu]

bug 538378 is the biggest issue; bug 644333 is the other main issue.

Comment 5

[Wayne Mery (:wsmwk)]

djst, can you test version 14 beta from http://www.mozilla.org/en-US/thunderbird/channel/ and tell us results?

Comment 6

[Dave Miller [:justdave]]

I just tried to delete 52,649 messages from a single folder (variety of subjects, not all alike) and I got a beachball for 36 seconds while trying to do "Select All" on the messages, and a beachball for 41 *minutes* when I actually hit Delete (which did a move-to-trash I think).  When it was done deleting them, memory usage was at 3.59 GB of physical RAM, which eventually shrank back to 2.5 GB or so after 5 or 10 minutes.

Comment 7

[Dave Miller [:justdave]]

I'm on Daily 12-07-05 build, Mac OS X 10.7.4

Comment 8

[David Tenser [:djst]]

Based on Dave's comment 6 and comment 7, it sounds like the bug wasn't fixed. It's still blocking the main UI.

Comment 9

[David :Bienvenu]

(In reply to David Tenser [:djst] from comment #8)
> Based on Dave's comment 6 and comment 7, it sounds like the bug wasn't
> fixed. It's still blocking the main UI.

We sped up this operation by an order of magnitude or so, but it's still slow for an extremely large number of messages.

Comment 12

[Wayne Mery (:wsmwk)]

perhaps bug 991658 is the same issue

Comment 14

[Wayne Mery (:wsmwk)]

Perhaps also some aspects of bug 842371 - main thread IO

For anyone who can reproduce, please run the performance profiler:

1. install profiler add-on into thunderbird - download the file from https://github.com/firefox-devtools/Gecko-Profiler-Addon/blob/master/gecko_profiler.xpi?raw=true and in Tools > add-ons click the gear to install add-on from file
2. Follow instructions at https://profiler.firefox.com/ (videos BASED ON FIREFOX at https://profiler.firefox.com/docs/#/./videos-intro )
3. Create a profiler URL and post it here.

You must be using Thunderbird beta from https://www.thunderbird.net/en-US/channel/ or current nightly build from https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/

Comment 15

[php4fan]

Come on, 8 years and still unfixed! (7 years since it's been known the attempted fix didn't work - not that it was any surprise given it didn't even attempt to address the main issue, which is point 1 below)

All the following should be obvious:

1. you don't do heavy work on the main thread blocking the whole UI of the application.
2. while the heavy work is done, you show a progress bar (or spinning icon) AND a cancel/abort button (whether aborting will result in a complete rollback or a partially done operation, you still give the option and warn the user accordingly. One always MUST be able to abort an operation that can take too long)

Additionally: This blocks both when you hit shift+Delete (let alone when you select all the messages, because of the stupid message summary in the message panel; that's tracked in another bug; I have already disabled the message panel), and then again when the prompt for confirmation of permanent deletion (without moving to trash) shows up and you click "Delete": the button doesn't even animate as if you hadn't pressed it. At least in the first case, there's no reason to do any heavy work (blocking or not) at all: between the time I hit shift+delete and the time a confirmation dialog shows up, there's nothing to be done. The moment I hit shift+delete, you don't even need to count how many messages I'm going to delete, let alone process them in any way, in order to show the confirmation prompt. Whatever it is that is taking that long, should be done after the confirmation.

"For anyone who can reproduce, please run the performance profiler"

Why do you need to wait for "anyone who can reproduce". It's not like this is something arcane that is difficult to reproduce or whose conditions to reproduce are unknown.

NOTE: I guess there's something wrong that makes the deletion of thousands of messages slow and that it shouldn't take this long in the first place; BUT even if you fix that part and speed this up, no matter how many orders of magnitude (well, unless you can make the deletion of millions of messages take just a few milliseconds), do NOT consider this fixed. This will be completely fixed when the deletion, even if takes just a few seconds, doesn't block the UI.

Actually, if the slowness itself is indeed excessive, then 1486641 is not really a duplicate.

Comment 16

[gene smith]

(In reply to David Tenser [:djst] from comment #0)

In my mozilla.com IMAP folders, I have one folder with (sometimes) over
15000 email messages. (These are nagios error emails that I rarely look at.)

When selecting all of those messages and hitting Delete, the UI completely
hangs for over a minute. This doesn't seem like the ideal way of handling
this. Maybe put the actual work in a separate worker thread and show a
progress bar instead?

Not trying to make excuses but a possible workaround for this might be to set tb to either "just mark as deleted" or "delete immediately" and then just delete the folder containing the 15k emails, then re-create the folder. (Of course, if there are emails in the folder you want to keep then this won't work.) Depending on the uid distribution of the emails, this may reduce the server load since only a simple imap delete of the folder/mailbox will occur.

(In reply to php4fan from comment #15)

Additionally: This blocks both when you hit shift+Delete (let alone when you select all the messages, because of the stupid message summary in the message panel; that's tracked in another bug; I have already disabled the message panel),

I'm working on another problem about copying large number of messages between imap server (bug 538375). I have a folder with 17k message and if I try to select them all (ctrl-a) it takes a looong time and there is no feedback that something is going on and, of course, the UI is blocked. I haven't tried disabling the message panel to see if that speeds it up.

and then again when the prompt for confirmation of permanent deletion (without moving to trash) shows up and you click "Delete": the button doesn't even animate as if you hadn't pressed it.

Not sure what you mean by "animate" here?

At least in the first case, there's no reason to do any heavy work (blocking or not) at all: between the time I hit shift+delete and the time a confirmation dialog shows up, there's nothing to be done. The moment I hit shift+delete, you don't even need to count how many messages I'm going to delete, let alone process them in any way, in order to show the confirmation prompt. Whatever it is that is taking that long, should be done after the confirmation.

I'll try this since not sure what you mean.

NOTE: I guess there's something wrong that makes the deletion of thousands of messages slow and that it shouldn't take this long in the first place; BUT even if you fix that part and speed this up, no matter how many orders of magnitude (well, unless you can make the deletion of millions of messages take just a few milliseconds), do NOT consider this fixed. This will be completely fixed when the deletion, even if takes just a few seconds, doesn't block the UI.

Changing the architecture so the UI is never blocked is probably hard (maybe for just me) due the mixture of c++ and .js style code. I'm sure when it was originally designed (15+ years ago) and coded, they only tested with a few message and it worked so they shipped it.

Comment 17

[php4fan]

I haven't tried disabling the message panel to see if that speeds it up.

I'm pretty sure it will. Before I did that, it would block three times: when I selected the messages, when I hit delete or shift+Delete, and then when clicking on the confirmation button. The blocking when selecting messages, I think, is due to the generation of the "summary" in the message panel, I remember a separate ages-old bug about that.

Not sure what you mean by "animate" here?

I mean the visual feedback you usually get when you click a button: where you see the button go "down" (i.e. into the screen) as if pressed, and then come "up" as if released. The UI apparently blocks as soon as you press the mouse button (actually I haven't tried pressing it and waiting before I release it) even before there's time to show the visual feedback of the button being pressed. That's an additional glitch on top of the main bug.

I'll try this since not sure what you mean

Let me rephrase slightly (I'll copy my own words and edit), let me know if it's clearer:

At least in the first instance (i.e. when you hit shift+Delete), there's no reason to do any heavy work at all: after I hit shift+delete and before a confirmation dialog shows up, TB should't need to count how many messages I'm going to delete, or process them in any way, in order to show the confirmation prompt. If I hit shift+Delete with N messages selected, showing the confirmation prompt should require O(1), not O(N). Only when I finally click on the confirmation button do I expect it to initiate an operation that will take O(N).

Comment 18

[gene smith]

Ok, I see what you are saying now. I didn't see the lack of "animation" on the delete key because with only a few messages it goes away so fast, I guess. I will try it later with a bigger group.

I never use the "delete" key when deleting emails since I just use the delete button. Also, I never use the shift-delete either. But when I did the shift delete-key it did work, but when I did a shift-delete-button it seems to ignore the shift key and deletes to trash. Not sure if that's how it should work or not.

Also, when you shift-delete-key tb really just marks the imap message with \deleted flag. So unless you have the non-default hidden pref "expunge on delete" set or are using an imap server that expunges on delete (like gmail), the messages are not really deleted from the mailbox until the folder is compacted. You can still get them back by setting tb to "just marked as deleted" and then undelete the messages that are crossed out. (Maybe undo will bring them back too, not sure.)

I'll have to check what happens when you hit shift-delete-key with N messages selected. It seems quite fast with just 3 or 4 like I just tried. I will try this with 17k later.
I also notice that it takes a long time if you ctrl-a select 17k messages when you want to copy them to another folder, then it takes the same amount of time again after you right-click to see to context menu to select where to copy to. During these waits there is no feedback or any confirmation that tb is working hard in the background. I think you might be able to click on other folders to terminate the activity if you get tired of waiting, but not sure. Also, if you are not sure you really did ctrl-a or right-click, if you do it again it may just start the process over again.

Comment 19

[php4fan]

Also, when you shift-delete-key tb really just marks the imap message with \deleted flag

I forgot to mention, I don't use imap, only pop3. So it's all local.
Again, I don't know whether or not the long time it takes to physically delete the messages is more than expected (it seems kind of excessive, but given the intrinsic efficiency of the historic file format, it just might be the best that can be done, I honestly don't know), what I know is that (a) the blocking of the main thread for all the time it takes to do the deletion is a bug, and (b) the wait (even if it were non-blocking) between hitting shift+delete and showing a confirmation prompt is also unjustified.

I never tried shift+delete_button, I wouldn't have expected it to work as shift+delete_key, that would be a nice feature.

Comment 20

[Wayne Mery (:wsmwk)]

(In reply to gene smith from comment #18)

...
I'll have to check what happens when you hit shift-delete-key with N messages selected. It seems quite fast with just 3 or 4 like I just tried. I will try this with 17k later.
I also notice that it takes a long time if you ctrl-a select 17k messages when you want to copy them to another folder, then it takes the same amount of time again after you right-click to see to context menu to select where to copy to. During these waits there is no feedback or any confirmation that tb is working hard in the background. I think you might be able to click on other folders to terminate the activity if you get tired of waiting, but not sure. Also, if you are not sure you really did ctrl-a or right-click, if you do it again it may just start the process over again.

Gene, Did you find anything interesting?

Comment 21

[gene smith]

(In reply to Wayne Mery (:wsmwk) from comment #20)

Gene, Did you find anything interesting?

I need to get back to behavior of activities with many messages, but I haven't seen anything new since I last posted to this bug.

Comment 22

[kamil]

When you will resolve this bug, you should probably consider real case emails, but not dummy emails. Usually, such large amounts of emails in single folder may appear in result of subscription to some blogs and news resources that generate tens/hundreds of emails per day, but only random 1-10% of these emails are worth to be read, after checking subjects. (At least in my case.)

So, what do we have?
a) these emails may contain HTML with large articles (several A4 pages of texts) and with abstracts of articles;
b) 90-99% of emails have status "not read".

Not sure that it will affect something, it is just a real test case.

Comment 24

[Wayne Mery (:wsmwk)]

mail maintenance/hygene is Sooo, so painful.

Waiting over 5 minutes now to delete 1k messages from gmail All Mail folder. TB Memory is up to 13gb.

Comment 25

[Wayne Mery (:wsmwk)]

Attachment: Sample of Thunderbird Daily-Not Responding bulk delete bug 693659.txt

I killed Thunderbird at 14gb of memory usage.
Mac spin dump attached

Comment 26

[Wayne Mery (:wsmwk)]

I had been deleting chunks of 300-500 messages at a time, even over a thousand. Although it did take a long time, sometimes 1-2 minutes. In each case, all the messages in the chunk had the same subject, etc.

For some reason this last chunk (I think it was just over 1k) just went haywire.