Open Bug 693659 Opened 13 years ago Updated 2 months ago

Main window completely locks up when trying to delete thousands of messages

Categories

(MailNews Core :: Backend, defect)

x86_64
All
defect
Not set
critical

Tracking

(Not tracked)

People

(Reporter: djst, Unassigned)

References

(Depends on 1 open bug, Blocks 2 open bugs)

Details

(Keywords: hang, perf, Whiteboard: [bulkoperations])

Attachments

(1 file)

In my mozilla.com IMAP folders, I have one folder with (sometimes) over 15000 email messages. (These are nagios error emails that I rarely look at.)

When selecting all of those messages and hitting Delete, the UI completely hangs for over a minute. This doesn't seem like the ideal way of handling this. Maybe put the actual work in a separate worker thread and show a progress bar instead?
this is a known bug, with several causes, and I've been looking into it recently. Two main issues are that some of the change notifications between c++ and js aren't batched, and there should be one undo action for this, instead of N where N is the number of messages.

Shift delete is probably quite a bit faster, if you don't care about the trash.
Component: Mail Window Front End → Folder and Message Lists
QA Contact: front-end → folders-message-lists
Thanks for the workaround tip; I'll try shift-delete next time. Oh, and sorry for filing a (I'm guessing?) dupe!
(In reply to David :Bienvenu from comment #1)
> this is a known bug, with several causes, and I've been looking into it

David what's the master bug for this ?
Keywords: perf
bug 538378 is the biggest issue; bug 644333 is the other main issue.
Depends on: 538378
djst, can you test version 14 beta from http://www.mozilla.org/en-US/thunderbird/channel/ and tell us results?
I just tried to delete 52,649 messages from a single folder (variety of subjects, not all alike) and I got a beachball for 36 seconds while trying to do "Select All" on the messages, and a beachball for 41 *minutes* when I actually hit Delete (which did a move-to-trash I think).  When it was done deleting them, memory usage was at 3.59 GB of physical RAM, which eventually shrank back to 2.5 GB or so after 5 or 10 minutes.
I'm on Daily 12-07-05 build, Mac OS X 10.7.4
Based on Dave's comment 6 and comment 7, it sounds like the bug wasn't fixed. It's still blocking the main UI.
(In reply to David Tenser [:djst] from comment #8)
> Based on Dave's comment 6 and comment 7, it sounds like the bug wasn't
> fixed. It's still blocking the main UI.

We sped up this operation by an order of magnitude or so, but it's still slow for an extremely large number of messages.
perhaps bug 991658 is the same issue
Blocks: 847285, 991658
Component: Folder and Message Lists → Backend
Depends on: 769346
Product: Thunderbird → MailNews Core
Whiteboard: [bulkoperations]
Severity: normal → major

Perhaps also some aspects of bug 842371 - main thread IO

For anyone who can reproduce, please run the performance profiler:

  1. install profiler add-on into thunderbird - download the file from https://github.com/firefox-devtools/Gecko-Profiler-Addon/blob/master/gecko_profiler.xpi?raw=true and in Tools > add-ons click the gear to install add-on from file
  2. Follow instructions at https://profiler.firefox.com/ (videos BASED ON FIREFOX at https://profiler.firefox.com/docs/#/./videos-intro )
  3. Create a profiler URL and post it here.

You must be using Thunderbird beta from https://www.thunderbird.net/en-US/channel/ or current nightly build from https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/

Severity: major → critical
OS: Windows 7 → All

Come on, 8 years and still unfixed! (7 years since it's been known the attempted fix didn't work - not that it was any surprise given it didn't even attempt to address the main issue, which is point 1 below)

All the following should be obvious:

  1. you don't do heavy work on the main thread blocking the whole UI of the application.
  2. while the heavy work is done, you show a progress bar (or spinning icon) AND a cancel/abort button (whether aborting will result in a complete rollback or a partially done operation, you still give the option and warn the user accordingly. One always MUST be able to abort an operation that can take too long)

Additionally: This blocks both when you hit shift+Delete (let alone when you select all the messages, because of the stupid message summary in the message panel; that's tracked in another bug; I have already disabled the message panel), and then again when the prompt for confirmation of permanent deletion (without moving to trash) shows up and you click "Delete": the button doesn't even animate as if you hadn't pressed it. At least in the first case, there's no reason to do any heavy work (blocking or not) at all: between the time I hit shift+delete and the time a confirmation dialog shows up, there's nothing to be done. The moment I hit shift+delete, you don't even need to count how many messages I'm going to delete, let alone process them in any way, in order to show the confirmation prompt. Whatever it is that is taking that long, should be done after the confirmation.

"For anyone who can reproduce, please run the performance profiler"

Why do you need to wait for "anyone who can reproduce". It's not like this is something arcane that is difficult to reproduce or whose conditions to reproduce are unknown.

NOTE: I guess there's something wrong that makes the deletion of thousands of messages slow and that it shouldn't take this long in the first place; BUT even if you fix that part and speed this up, no matter how many orders of magnitude (well, unless you can make the deletion of millions of messages take just a few milliseconds), do NOT consider this fixed. This will be completely fixed when the deletion, even if takes just a few seconds, doesn't block the UI.

Actually, if the slowness itself is indeed excessive, then 1486641 is not really a duplicate.

(In reply to David Tenser [:djst] from comment #0)

In my mozilla.com IMAP folders, I have one folder with (sometimes) over
15000 email messages. (These are nagios error emails that I rarely look at.)

When selecting all of those messages and hitting Delete, the UI completely
hangs for over a minute. This doesn't seem like the ideal way of handling
this. Maybe put the actual work in a separate worker thread and show a
progress bar instead?

Not trying to make excuses but a possible workaround for this might be to set tb to either "just mark as deleted" or "delete immediately" and then just delete the folder containing the 15k emails, then re-create the folder. (Of course, if there are emails in the folder you want to keep then this won't work.) Depending on the uid distribution of the emails, this may reduce the server load since only a simple imap delete of the folder/mailbox will occur.

(In reply to php4fan from comment #15)

Additionally: This blocks both when you hit shift+Delete (let alone when you select all the messages, because of the stupid message summary in the message panel; that's tracked in another bug; I have already disabled the message panel),

I'm working on another problem about copying large number of messages between imap server (bug 538375). I have a folder with 17k message and if I try to select them all (ctrl-a) it takes a looong time and there is no feedback that something is going on and, of course, the UI is blocked. I haven't tried disabling the message panel to see if that speeds it up.

and then again when the prompt for confirmation of permanent deletion (without moving to trash) shows up and you click "Delete": the button doesn't even animate as if you hadn't pressed it.

Not sure what you mean by "animate" here?

At least in the first case, there's no reason to do any heavy work (blocking or not) at all: between the time I hit shift+delete and the time a confirmation dialog shows up, there's nothing to be done. The moment I hit shift+delete, you don't even need to count how many messages I'm going to delete, let alone process them in any way, in order to show the confirmation prompt. Whatever it is that is taking that long, should be done after the confirmation.

I'll try this since not sure what you mean.

NOTE: I guess there's something wrong that makes the deletion of thousands of messages slow and that it shouldn't take this long in the first place; BUT even if you fix that part and speed this up, no matter how many orders of magnitude (well, unless you can make the deletion of millions of messages take just a few milliseconds), do NOT consider this fixed. This will be completely fixed when the deletion, even if takes just a few seconds, doesn't block the UI.

Changing the architecture so the UI is never blocked is probably hard (maybe for just me) due the mixture of c++ and .js style code. I'm sure when it was originally designed (15+ years ago) and coded, they only tested with a few message and it worked so they shipped it.

I haven't tried disabling the message panel to see if that speeds it up.

I'm pretty sure it will. Before I did that, it would block three times: when I selected the messages, when I hit delete or shift+Delete, and then when clicking on the confirmation button. The blocking when selecting messages, I think, is due to the generation of the "summary" in the message panel, I remember a separate ages-old bug about that.

Not sure what you mean by "animate" here?

I mean the visual feedback you usually get when you click a button: where you see the button go "down" (i.e. into the screen) as if pressed, and then come "up" as if released. The UI apparently blocks as soon as you press the mouse button (actually I haven't tried pressing it and waiting before I release it) even before there's time to show the visual feedback of the button being pressed. That's an additional glitch on top of the main bug.

I'll try this since not sure what you mean

Let me rephrase slightly (I'll copy my own words and edit), let me know if it's clearer:

At least in the first instance (i.e. when you hit shift+Delete), there's no reason to do any heavy work at all: after I hit shift+delete and before a confirmation dialog shows up, TB should't need to count how many messages I'm going to delete, or process them in any way, in order to show the confirmation prompt. If I hit shift+Delete with N messages selected, showing the confirmation prompt should require O(1), not O(N). Only when I finally click on the confirmation button do I expect it to initiate an operation that will take O(N).

Ok, I see what you are saying now. I didn't see the lack of "animation" on the delete key because with only a few messages it goes away so fast, I guess. I will try it later with a bigger group.

I never use the "delete" key when deleting emails since I just use the delete button. Also, I never use the shift-delete either. But when I did the shift delete-key it did work, but when I did a shift-delete-button it seems to ignore the shift key and deletes to trash. Not sure if that's how it should work or not.

Also, when you shift-delete-key tb really just marks the imap message with \deleted flag. So unless you have the non-default hidden pref "expunge on delete" set or are using an imap server that expunges on delete (like gmail), the messages are not really deleted from the mailbox until the folder is compacted. You can still get them back by setting tb to "just marked as deleted" and then undelete the messages that are crossed out. (Maybe undo will bring them back too, not sure.)

I'll have to check what happens when you hit shift-delete-key with N messages selected. It seems quite fast with just 3 or 4 like I just tried. I will try this with 17k later.
I also notice that it takes a long time if you ctrl-a select 17k messages when you want to copy them to another folder, then it takes the same amount of time again after you right-click to see to context menu to select where to copy to. During these waits there is no feedback or any confirmation that tb is working hard in the background. I think you might be able to click on other folders to terminate the activity if you get tired of waiting, but not sure. Also, if you are not sure you really did ctrl-a or right-click, if you do it again it may just start the process over again.

Also, when you shift-delete-key tb really just marks the imap message with \deleted flag

I forgot to mention, I don't use imap, only pop3. So it's all local.
Again, I don't know whether or not the long time it takes to physically delete the messages is more than expected (it seems kind of excessive, but given the intrinsic efficiency of the historic file format, it just might be the best that can be done, I honestly don't know), what I know is that (a) the blocking of the main thread for all the time it takes to do the deletion is a bug, and (b) the wait (even if it were non-blocking) between hitting shift+delete and showing a confirmation prompt is also unjustified.

I never tried shift+delete_button, I wouldn't have expected it to work as shift+delete_key, that would be a nice feature.

(In reply to gene smith from comment #18)

...
I'll have to check what happens when you hit shift-delete-key with N messages selected. It seems quite fast with just 3 or 4 like I just tried. I will try this with 17k later.
I also notice that it takes a long time if you ctrl-a select 17k messages when you want to copy them to another folder, then it takes the same amount of time again after you right-click to see to context menu to select where to copy to. During these waits there is no feedback or any confirmation that tb is working hard in the background. I think you might be able to click on other folders to terminate the activity if you get tired of waiting, but not sure. Also, if you are not sure you really did ctrl-a or right-click, if you do it again it may just start the process over again.

Gene, Did you find anything interesting?

Flags: needinfo?(gds)

(In reply to Wayne Mery (:wsmwk) from comment #20)

Gene, Did you find anything interesting?

I need to get back to behavior of activities with many messages, but I haven't seen anything new since I last posted to this bug.

Flags: needinfo?(gds)

When you will resolve this bug, you should probably consider real case emails, but not dummy emails. Usually, such large amounts of emails in single folder may appear in result of subscription to some blogs and news resources that generate tens/hundreds of emails per day, but only random 1-10% of these emails are worth to be read, after checking subjects. (At least in my case.)

So, what do we have?
a) these emails may contain HTML with large articles (several A4 pages of texts) and with abstracts of articles;
b) 90-99% of emails have status "not read".

Not sure that it will affect something, it is just a real test case.

Depends on: 1242042

mail maintenance/hygene is Sooo, so painful.

Waiting over 5 minutes now to delete 1k messages from gmail All Mail folder. TB Memory is up to 13gb.

I killed Thunderbird at 14gb of memory usage.
Mac spin dump attached

I had been deleting chunks of 300-500 messages at a time, even over a thousand. Although it did take a long time, sometimes 1-2 minutes. In each case, all the messages in the chunk had the same subject, etc.

For some reason this last chunk (I think it was just over 1k) just went haywire.

Keywords: hang
See Also: → 538375
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: