528713 - Drag&Drop in IMAP in Shredder 2009114, causes application freeze / lock up / unresponsive

Status: RESOLVED INCOMPLETE

(Thunderbird :: Folder and Message Lists, defect)

Description

[Thomas Harold]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091114 Shredder/3.0.1pre

The delay between the start of the drag-n-drop operation and the change of the cursor to an arrow with box becomes longer and longer as you select more messages.  I'm not timing the actual move operation, but merely the UI responsiveness after a user selects N messages and then uses the cursor to begin the drag-n-drop operation to another folder.

Time is from the start of the drag until the cursor changes shape.  During this time period, TB becomes unresponsive and appears to be hung.

1250 messages ~2 seconds
1600 messages ~3 seconds
1700 messages ~6 seconds
1850 messages ~8-9 seconds
2000 messages ~14 sec
2100 messages ~18 sec
2200 messages ~20-25 seconds
2300 messages ~28 sec
3100 messages ~95 sec

What we have here is what looks like an O(N^2) operation, or at least something goes horribly wrong after more then ~2000 messages being selected for the drag-n-drop move operation.  

In the TB 3 Beta 3 build, the cursor change was immediate, even with 5700 messages selected for the drag-n-drop move operation.

Reproducible: Always

Steps to Reproduce:
1. Take an IMAP folder with more then a few thousand messages
2. Select all (Ctrl-A)
3. Attempt to drag and drop move them to another IMAP folder
Actual Results:  
The TB/Shredder UI will freeze and become completely unresponsive.  The cursor never changes to a move/copy type cursor (with the box or plus sign).  Thunderbird.exe starts to use 100% of the CPU core that it is running on, memory usage stays stable.  All background processing of other TB tasks completely stops (such as emails being downloaded in the background).

Expected Results:  
The cursor should change to an arrow with a box to show that a move operation will occur if you release the left mouse button.  Prior to the latest builds, the cursor would change shape almost immediately after beginning the drag-n-drop and moving the cursor over a valid destination folder.

This seems to have started with the various RC 1 builds that I have been trying.  The following build also exhibits the same problem:

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091112 Thunderbird/3.0

The following builds do NOT exhibit the same issue:

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3pre) Gecko/20090223 Thunderbird/3.0b2

Comment 1

[Wayne Mery (:wsmwk)]

Thomas, it would greatly help if you could either narrow the regression range [1] down to a very short period like a day or two (from 20090223-20091112) or if someone has debug suggestions you can try. [1] http://www.rumblingedge.com/2009/02/24/howto-find-regression-windows-through-manual-binary-search/

The underlying issue is performance. unless I am missing something, none of the open 3.0/trunk perf move/copy bugs for the period are marked regression, and no open copy/move regression bugs describe any bulk action or performance issue. However, allowing for imperfect queries and bug reports, this might still be a reported bug. 

open perf move/copy bugs
https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short_desc_type=anywordssubstr&short_desc=cop+mov+drag&product=MailNews+Core&product=Thunderbird&resolution=---&bug_severity=critical&bug_severity=major&chfieldto=Now&field0-0-0=short_desc&type0-0-0=substring&value0-0-0=slow+perf&field0-0-1=keywords&type0-0-1=substring&value0-0-1=perf

all move/copy regressions
https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short_desc_type=anywordssubstr&short_desc=cop+mov+drag&product=MailNews+Core&product=Thunderbird&version=Trunk&version=unspecified&version=1.9.0+Branch&version=3.0&version=Trunk&version=1.9.1+Branch&version=1.9.2+Branch&keywords_type=allwords&keywords=regression&resolution=---&bug_severity=critical&bug_severity=major&bug_severity=normal&bug_severity=minor&chfieldfrom=2009-02-23&chfieldto=Now&chfield=%5BBug+creation%5D

Comment 2

[Thomas Harold]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4pre) Gecko/20090915 Thunderbird/3.0b4
- Cursor change is immediate, even when selecting ~6200 messages and attempting to drag-drop to another folder in IMAP.  So it worked fine as of the Sep 15th Beta 4 build.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091112 Thunderbird/3.0
- Exhibits the same problem as the original complaint.  Small selections work fine, but the time required for the cursor changes increases rapidly once you get past 1800-2000 messages (2.5GHz AMD Phenom quad core).

I'm trying to find other Win32 installers from the October window.  Links to download the nightly Win32 installers would be helpful.

My initial guess is that it's related to the new preview / message pane window which attempts to summarize.  Except that even when I turn off the message pane in the View menu, the problem still occurs.

Comment 3

[Thomas Harold]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5pre) Gecko/20091016 Shredder/3.0pre
- Works fine, cursor change is immediate.

Comment 4

[Thomas Harold]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6pre) Gecko/20091105 Shredder/3.0pre
- Same issue as original complaint.  Selection sizes over 1500-2000 messages which are then dragged to another folder result in longer and longer time before the cursor changes to a "move" cursor.

Comment 5

[Thomas Harold]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5pre) Gecko/20091029 Shredder/3.0pre
- Same issue as original complaint.  That puts the date range where this problem started in the 20091016 to 20091029 range.

Worse... when TB locks up the UI like this, it also affects Mozilla Firefox 3.5 (current version).  Killing the thunderbird.exe process does not fix the firefox.exe process.  Firefox UI remains locked up and has to be killed off.  I've had it happen quite a few times while testing various versions of Thunderbird/Shredder.  I've no idea how to log that as a bug report as it affects both products.

Comment 6

[David :Bienvenu]

This is a known issue, and we've been landing fixes on various branches so that the next release off that branch will have the fix. See bug 514148 - I believe this is a dup of that basic issue, though there may also be a specific issue about TB.

Comment 7

[Thomas Harold]

I'm pretty sure that this is a basic issue with Thunderbird's UI and not a duplicate of bug 514148.  The issue at hand here is why does TB lock up for longer and longer periods as you select more and more messages and then attempt to drag & drop them to another IMAP folder.

The cross-application bug was mentioned as a side-issue to the original problem. (The cross-application bug is definitely bug 514148 however).

Bug 514148 showed up in early September 2009.  The bug that I'm seeing in TB with regards to the drag-and-drop taking more and more time to change the cursor to a "move" cursor did not show up until sometime between Oct 16 and Oct 29.  That's about the time that the message pane was changed to attempt to summarize the selection when you select multiple messages.

Comment 8

[Thomas Harold]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.3a1pre) Gecko/20091116 Shredder/3.1a1pre
- Better, but still not fixed.  Attempting to drag-n-drop 3300 messages from one folder to another results in UI lockup of over 12 seconds.

2600 msgs ~10 sec
3300 msgs ~12 sec
4100 msgs ~15 sec

This issue seems to affect both IMAP source/destination folders as well as local folders.  So it's not just an IMAP issue.

On the upside, the time required to start the drag-n-drop operation is not O(N^2) any longer, it's much closer to O(N).  I just wish it was faster and more immediate like it was back prior to Oct 15th.

Comment 9

[Thomas Harold]

However, even with Shredder 3.1a1pre (1.9.3), TBird eventually pops up the error "Warning: Unresponsive script" once I try to drag over 6000 messages at the same time.

Script: chrome://communicator/content/utilityOverlay.js:181

The problem is that I have to move the mouse to deal with this prompt, so now I don't know where my messages are going to be moved to.

Comment 10

[Wayne Mery (:wsmwk)]

(In reply to comment #5)
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5pre) Gecko/20091029
> Shredder/3.0pre
> - Same issue as original complaint.  That puts the date range where this
> problem started in the 20091016 to 20091029 range.

excellent work. however, there are 76 checkins in that time period.
can you narrow the range to 1/4?

Comment 11

[Thomas Harold]

Definitely still broken in TB 3.0 RC 1 build 2 (from Nov 17th), but *somewhat* fixed in the 1.9.3 line (still basically broken since the script times out).  I'm going back into October to see if I can narrow down when it first showed up.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5pre) Gecko/20091025 Shredder/3.0pre
- Broken

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5pre) Gecko/20091022 Shredder/3.0pre
- Broken

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5pre) Gecko/20091020 Shredder/3.0pre
- Broken

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5pre) Gecko/20091018 Shredder/3.0pre
- Broken

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5pre) Gecko/20091017 Shredder/3.0pre
- Broken

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5pre) Gecko/20091016 Shredder/3.0pre
- Works fine

So that narrows it down to pretty much a specific day (Oct 17th) for the 1.9.1 tree.  I tested the above by attempting to drag-n-drop 2000 or 4000 messages between *local* folders.  The 2000 message selection usually took 2-3 sec to change the cursor, while the 4000 message selection took > 30 sec and I didn't bother to wait and time it.  Based on earlier tests, it would've taken a few minutes.

Comment 12

[Wayne Mery (:wsmwk)]

most notable landings on oct 16/17-early-am are
Bug 227305 - "Support drag-drop single message to desktop / file-system window (e.g. Explorer)" [r+a=bienvenu]
 handle failure to stream offline imap message during compact, r/sr=standard8, possible fix for 509748, a=blocker 
Bug 459693 - Eliminate nsFileSpec and nsIFileSpec (references) from MailNews;  (Hv1) Bug 33451 <nsOutlookMail.cpp> followup

Comment 13

[Andrew Sutherland [:asuth] (he/him)]

Yeah, it's definitely the fix from bug 227305.  It's O(n) to generate a unique temporary file name for O(n) messages, so O(n^2).  Not to mention it is getting URIs from gFolderDisplay and then using the URI service to turn them back into headers when it could just get the message headers directly from gFolderDisplay in the first place.

Comment 14

[David :Bienvenu]

A js hashtable/array would speed this up quite a bit.

Comment 15

[Thomas Harold]

Testing RC1 build 3 - Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091121 Thunderbird/3.0

Times listed are the time between I attempt to drag the selected messages with the left-mouse button until the cursor changes into the "pending move" type cursor (arrow + small grey box) over the destination folder.

Between local folders
1600 msgs ~5 sec
1775 msgs ~7 sec
1850 msgs ~20 sec
1950 msgs ~15 sec
2150 msgs ~21 sec
2250 msgs ~23 sec
2550 msgs ~36 sec

Between IMAP folders
1750 msgs ~5 sec
1850 msgs ~5 sec
2000 msgs ~5 sec

I'm going to do some more testing later on to see whether things are truly different between IMAP folders and Local folders or whether it was a temporary condition.

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> What we have here is what looks like an O(N^2) operation, 

Sounds phenomenon of Bug 452221(occurs on both IMAP & local mail folder) is involved.

(In reply to comment #8)
> On the upside, the time required to start the drag-n-drop operation is not
> O(N^2) any longer, it's much closer to O(N).

Bug 530461 has been opened for such IMAP only issue.

Comment 17

[Wayne Mery (:wsmwk)]

(In reply to comment #16)
> (In reply to comment #0)
> > What we have here is what looks like an O(N^2) operation, 
> 
> Sounds phenomenon of Bug 452221(occurs on both IMAP & local mail folder) is
> involved.
> 
> (In reply to comment #8)
> > On the upside, the time required to start the drag-n-drop operation is not
> > O(N^2) any longer, it's much closer to O(N).
> 
> Bug 530461 has been opened for such IMAP only issue.

Thomas, please update bug with your thoughts on this and bug 530461 comment 4, so we can consolidate the issues. Thanks.

Comment 18

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #15)
> Between local folders
> 1600 msgs ~5 sec
> 1775 msgs ~7 sec
> 1850 msgs ~20 sec
> 1950 msgs ~15 sec
> 2150 msgs ~21 sec
> 2250 msgs ~23 sec
> 2550 msgs ~36 sec

If write of mail data to local mail folder occurs, Bug 539389 happens, and performance issue is exposed if many mails are copied even if writing to local HDD instead of network drive.
"Drag&Drop of mails within same account" consists of (a) "copy mails to target folder" and (b) "delete mails from source folder".
(a) is linear to total mail data size, but (b) is not linear to number of mails nor total mail data size.
Bug opener, please isolate problem of (a) and (b), and please keep "one problem per a bug".

Comment 19

[Joshua Cranmer [:jcranmer]]

RESOLVED INCOMPLETE due to lack of response to previous question. If you feel this change was made in error, please respond to this bug with your reasons why.