1100940 - Moving a bunch of messages from a folder to another (pop3 local) is insanely slow

Status: RESOLVED INCOMPLETE

(MailNews Core :: Database, defect)

Description

[teo8976]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36

Steps to reproduce:

- Went to a folder of a pop3 account
- clicked on a message
- shift+clicked on the last message, selecting a total of more than 200,000 messages
- moved them to a subfolder by dragging and dropping


Actual results:

1 - The very act of selecting the messages took a few seconds. 

2 - After they were selected, being able to drag them also took a few seconds. I mean, the time that elapsed between when I pressed the button on the selected message and when I started seeing the icon that indicates I was dragging something. 

And the most important of all:
3 - actually moving the messages took HOURS (literally, several hours). 


Expected results:

1 - That should be instantaneous, as there's no noeed whatsoever to perform ANY action on every single message when you're merely doing a selection of a bunch of _contiguous_ messages. If the mere action of indicating "I want to select from message M to message N" implies doing some processing per-message, something is badly designed.

2 - No matter how many messages are selected, no matter whether contiguous or not, starting a drag operation is merely a UI thing, and the feedback should be immediate and should not depend on the number of messages selected, in any way. Whatever needs to be computed should be computed when you drop, *not* when you start dragging.



3 - It MUST be taken into account that these were contiguous messages. If I, as a human being, were to open the text files for the source and target message folder, carefully figure out where the first selected message started and the last ended, and the point in the target file where to move, and manually do the edits in Vim and save the files, it would have taken less time than doing this through TB. This is insane. Given that they are contiguou, TB could and hence should calculate the portion of file to be copied and copy it as a block. This could be done in a few seconds at most. (these were all small messages of a few hundred characters each and with no attachments: the amount of data being moved was in the order of tens of MB). 
And even assuming the messages are processed one at a time, the time it took still seems disproportioned

Comment 1

[:aceman]

1. Even the selection of messages is an expensive task because we build a summary of the message contents in the message preview pane. Should be fixed in bug 778907 (TB36)

2. Dragging is also expensive as we must prepare some data for the potential drop of the messages later on. Should have been fixed in bug 812923 (TB24). If that is not the case for 200 000 messages, please file that separately as that is a completely different component (UI) from the moving backend.

3. Your description of "manual" copying of the messages is not complete. TB also needs to move the message metadata (size, some header fields, status) to the message database (msf file) of the destination folder. I assume that part takes most of the time in this as we need to build the DB transactions. You could simulate that by copying the message data as you describe to the new folder-file, then removing it's .msf file. Then see how long TB builds the index (msf) for that folder when starting up.

In the mean time, try to copy such big numbers of messages by selecting them, then rightlick and use "Move to folder".

Comment 2

[WADA:World Anti-bad-Duping Agency]

(In reply to teo8976 from comment #0)
> Steps to reproduce:
> - Went to a folder of a pop3 account
> - clicked on a message
> - shift+clicked on the last message, selecting a total of more than 200,000 messages
> - moved them to a subfolder by dragging and dropping

"Move mails from local FolderA to local FolderB" consists of "Copy mails from FolderA to FolderB" + "Delete mails at FolderB".
So, above operation consists of:
   1. Select 200,000 mails at Thread Pane(messae list).
        If move by Drag&Drop, "generating file name for 200,000 mails" will be needed upon DragStaart,
        in case of "Save mail data at somewhere by Drg&Drop".
   2. Copy 200,000 mails from FolderA to FolderB == Do following for 200,000 mails :
             (2-1) Generate MsgDBHdr of copied mail at FolderB
             (2-2) Read entire mail data of a mail held in file named FolderA(not FolderA.msf)
             (2-3) Append entire mail data of the to file named FolderB(not FolderB.msf)
   3. Delete 200,000 mails from FolderA == Do following for 200,000 mails :
             (3-1) Hold data for "Undo" in memory(in JavScript object)
             (3-2) Same action as "Shift+Delete" : 
                       (3-2-a) Write "Deleted flag" in X-Mozill-Status: for a mail in file named FolderA.
                       (3-2-b) Remove MsgDBHdr of a mail.

> And the most important of all:
> 3 - actually moving the messages took HOURS (literally, several hours). 

Because of "performance issue", merely shouting "Slow!!!", "Takes too lo---ng!" etc. is useless at B.M.O where place to report bug to developers(never support forum).
Quontitive data is mandatory for problem analysis by developers.

How long does it take at which step of above? Is it O( number of mail ) Or ( O N^2 ).
How about memory consumption by Tb? (both Virtual Memory and Real Memory).
If Swapping occur in MS Windows, and if Ms Windows expands swap file, it takes pretty long. This is design of MS Win.
Sufficiently large swap file dize is correctly used by you?

In any step of above, performance issue is already known, athough some are already resolved.
   For example, in step (2-3), API level Write is requested for each line of mail data. No buffering is currently used.
   Because "Disk Cache" is utilzed by OS, if local HDD, impact is not so high.
   However, if network file, "write request per line" will surely produces performance issue.
To resolve such problems, "one bug per a problem" is needed.
Open separate bug for performance issue in each step of above, please.

Comment 3

[WADA:World Anti-bad-Duping Agency]

FYI.

(In reply to teo8976 from comment #0
> And the most important of all:
> 3 - actually moving the messages took HOURS (literally, several hours). 

Example of "critical performance issue" in "similar to Shift+Delete" step of "move many mails" : Bug 452221
   mail#1 is root of  thread. mail#2 to mail#Z has same subject, so all is in same thread.
   delete mail#1 by move mails => Because thread root is deleted, thread root is changed to mail#2.
        => Parent of mail#3 to mail#Z is changed to mail#2.
   delete mail#(N-1) by move mails => Because thread root is deleted, thread root is changed to mail#N
        => Parent of mail#(N+1) to mail#Z is changed to mail#(N+1)
   It continues until all mails are deleted at move source folder.
In test for Bug 452221, severe swapping and Swap file expansion by Win always occurred at order of 1000 mails, so phenomenon was "move mail" won't complete, and available method to recover was "Re-boot PC" only :-)
This may be already improved by some solutions such as "remove from bottom" instead of "remove from top", but I'm not sure.

Another example : "Mark As Junk" of many mails.
   High memory consumption and slowness is observed at far small number of mails than "move mails",
   because memory resources consumpsio is larger in Junk Filter,
   and similar performance issue to "move mail" can be observed because "Mark as Junk" invokes "Junk Move"=="Move to Junk".

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to teo8976 from comment #0)
> 2 - No matter how many messages are selected, no matter whether contiguous or not, starting a drag operation is merely a UI thing, 
> and the feedback should be immediate and should not depend on the number of messages selected, in any way.

Do you understand that Tb unfortunately has mode of "mail.operate_on_msgs_in_collapsed_threads=true"? :-)
Tb does do excess work by default upon mail selection in addition to mandatory "job of mail selection".

> Whatever needs to be computed should be computed when you drop, *not* when you start dragging.

Do you understand that "who works upon Drop at Desktop" is never Thunderbird?
Do you understand that "resources relevant to the Drag&Drop" should be listed upon DragStart event and the list of resources should be passed to DragOver event handler and Drop event handler?

(a) When Drag&Drop of mails, "list of resources relevant to the Drag&Drop" is list of MsgDBHdrs for Drop at folder.
(b) For "Drop at DeskTop", file name for each mail should be provided by DragStart event handler,
      and the DragStart event handler is Thunderbird.
      And, file name passed to Windows Explorer should be unique.
Both (a) and (b) should be executed by DragStart event handler upon DragStart event.

Unfortunately, "Drag&Drop of multiple items from Tb to Windows Explorer" is not supported yet.
So, IIRC, Tb currently doesn't do job of (b) upon DragStart event.(generates one file name of first selected mail only).
Tb may already do job of (b) for future "Drag&Drop of multiple items from Tb to Windows Explorer" support.
IIRC, Tb's algorithm for generating unique file name is not so well, and it's perhaps O(n*2) in worst case.
However, as  :aceman says, I believe fix of bug 812923 is still effective.
Did you actually see phenomenon of  bug 812923 in your test?
What you saw upon "selecting 200,000 mails" is "slowness due to mail.operate_on_msgs_in_collapsed_threads=true", isn't it?
Do you see slowness in "selecting 200,000 mails" with mail.operate_on_msgs_in_collapsed_threads=false?
(mail.operate_on_msgs_in_collapsed_threads=false is effective solution of Bug 778907)

Comment 5

[WADA:World Anti-bad-Duping Agency]

To bug opener:
Do following, please, to see "pure performance issue in move many mails".
(1) mail.operate_on_msgs_in_collapsed_threads=false, and select 200,000 mails,
     to rule out issues in "mail selection".
(2) Copy the selected 200,000 mails via menu instead of Drag&Drop,
      to see performance issue in "Copy Phase of Move", to rule out issues in Drag&Drop.
(3) Shift+Delete of all selected mails, 
      to see performance issue in "Delete Phase of Move", to rule out issues due to resources for "Undo Move".
      Note: "Copy to Trash" is skipped when "Shift+Delete", and "Undo of Shift+Delete" is not supported.
                  So, confirmation dialog is shown by Tb when Shift+Delete.

Is step (2) of above O( N )? Or O( N^2 )? Or O( N * logN )?
Is step (3) of above O( N )? Or O( N^2 )? Or O( N * logN )?

(In reply to teo8976 from comment #0)
> 3 - It MUST be taken into account that these were contiguous messages. 

In "Copy mails from FolderA to FolderB", "Read data from file/write data  to file" by Tb is approximately:
   for(each MsgDBHdr of selected mails) {
      (1) Read data of FolderA at Offset=messageOffset, length=messageSize, with 4KB buffer.
      (2) Write(append mode) each line in buffer(delimited by CRLF) to FolderB
            This is Bug 539389. Bug 769346 is for "Bug 539389 has come back" or "Zombie of Bug 539389 appeared".
   }
   Note: messageOffset=messageKey usually if local mail folder, and messageKey is shown as "Order Received" column value.

Because "Seek to messageOffset of a mail" happens upon each "(1) Read to buffer" of above, "Mails are sorted by Order Received column"=="sorted by messageOffset" may affect on performance.
However, "Seek to EOF of FolderB on same HDD" happens sooner or later, so  "Seek to messageOffset of a mail at FolderA" usually occurs upon each "Read to Disk-Cache with buffer size in OS" for some "(1) Read to buffer at FolderA".
I believe key factor in above is "over head of API call for each line" in step (2).
I believe "Required seek at FolderA due to Read of not-consecutive Offset" is never key factor.
I guess "Sorting by Offset before copy multiple mails" is not so effective in performance gain.

Do you see difference between next?
(i)   Sort by Order received column in ascending order, select second mail, Shift+select of bottom mail, Copy to oher folder.
(ii)  Sort by Order received column in descending order, select second mail, Shift+select of bottom mail, Copy to oher folder.
(iii) Sort by Subject, select second mail, Shift+select of bottom mail, Copy to other folder.
(I don't know Tb's behavior when all is selected. Tb may do "copy entire file" if "all is selected" and "deleted bytes=0"(no deleted mail, no need of Compact). So, don't select one mail.)

For "It MUST be taken into account that these were contiguous messages.".
You perhaps say "if continuous small mails are copied, issue "one read API for multiple mails" 

Buffer size for read=4KB
If size of each mail is one byte, "Number of Read API call" can be reduced to 1/4000 by it.
But even if small mail, total mail size is usually near to or larger than 1KB.
(Received: header is long, and many Received: is usually appended).
So, "Number of Read API call" can be reduced to 1/3 to 1/2 by it in best case.
Is performance gain by it so large?

As you say, "sequential read with large buffer" is a good way for performance gain.
But I believe major cause of performance issue in "Copy mails between local folder" is not in "Read data from file" part.
I believe it's in "One write request for each line" part.

And, if msgstore/maildirstore(one file per a mail), "these were contiguous messages" is a kind of nonsense.

Comment 6

[matteo sisti sette]

> Do you understand that Tb unfortunately has mode 
> of "mail.operate_on_msgs_in_collapsed_threads=true"? :-)

No I don't. I'm not even sure what a collapsed thread is. I kind of guess what it is supposed to be, but in any folder listing I've never seen anything that looks like a "thread". I do know I can do right-click on a message and do  "open message in conversation", and see the thread it belongs to. But messages in Inbox and Sent and other folders are only shown as individual messages (and I like it that way, not the Gmail webmail way where threads are listed). So I'm confused that when selecting a bunch of messages (from a list that lists individual messages) any work related to threads needs to be done.


> Do you understand that "who works upon Drop at Desktop" is never Thunderbird?
> Do you understand that "resources relevant to the Drag&Drop" should be listed 
> upon DragStart event and the list of resources should be passed to DragOver 
> event handler and Drop event handler?

Ok, I guess you mean in order to interact with the OS (in case, for example, you drag messages and drop them outside of TB). I haven't thought about that. However:
You must take into account that most of the times, when one drags something from within TB, one is going to drop it within TB. Especially because:
- Currently, dragging a message from TB to anywhere else is COMPLETELY USELESS (I've just tried doing that to the desktop, and it only creates some sort of link. It could be of some use if it saved a copy of the message or something).
- In the case of Ubuntu/Unity, the drag-n-drop between applications is particularly ****, rendering it useless most of the times (e.g. you drag an item from an application's window, which you could perfectly drop into another application's window, and that would actually work, but the desired target's icon in the Launcher doesn't show up as a suitable target, though it it).

Given that, if conforming to the OS's standards for dragging and dropping items across programs imposes burdens that degrade performance in such a terrible way (and it should be investigated to which point's that's really unavoidable) even when dragging and dropping within the application, then it should be considered whether it is worth doing that, or instead managing drag-n-drop only internally until, for example, one _actually_ crosses the window's border, or give an option to enable/disable that kind of interaction, or something. 

Anyway, what is certainly unacceptable is that you start dragging and have no feedback whatsoever until all the information is collected. At the very, very least, you should have a "busy"-icon cursor show up while the information is collected and the drag operation is actually started, and NOT block the UI.


> What you saw upon "selecting 200,000 mails" is "slowness due to 
> mail.operate_on_msgs_in_collapsed_threads=true", isn't it?
> Do you see slowness in "selecting 200,000 mails" 
> with mail.operate_on_msgs_in_collapsed_threads=false?

I don't even know how I am suppose to change that setting. A user shouldn't be supposed to tinker with settings in order to get a decent performance in normal everyday use of the program.

Comment 7

[Wayne Mery (:wsmwk)]

(In reply to matteo sisti sette from comment #6)
> > Do you understand that Tb unfortunately has mode 
> > of "mail.operate_on_msgs_in_collapsed_threads=true"? :-)
> 
> No I don't. I'm not even sure what a collapsed thread is. I kind of guess
> what it is supposed to be, but in any folder listing I've never seen
>... 
> > What you saw upon "selecting 200,000 mails" is "slowness due to 
> > mail.operate_on_msgs_in_collapsed_threads=true", isn't it?
> > Do you see slowness in "selecting 200,000 mails" 
> > with mail.operate_on_msgs_in_collapsed_threads=false?
> 
> I don't even know how I am suppose to change that setting. A user shouldn't
> be supposed to tinker with settings in order to get a decent performance in
> normal everyday use of the program.

He's asking you to test it, not live with it.
Chanage prefs is easy http://lmgtfy.com/?q=thunderbird+preference+change

Comment 8

[teo8976]

Ok I see. I have no time to perform that test, though.
If you are interested in fixing the bug, consider whether you actually need _me_ to provide that info, because unfortunately I'm afraid that's not going to happen. (I don't intend to be mean, just saying things as they are, in case this report remains in the "needsinfo" state forever)

Comment 9

[Wayne Mery (:wsmwk)]

In that case, wada can make the dupe (or reopen and confirm) if he feels he knows for sure what the exact match/issue is.  Otherwise, without your testing there's no way for us to know.

Comment 10

[Wayne Mery (:wsmwk)]

I'm not convinced this will be helped by bug 11050. If it tunrs out to not help, and you are willing to provide more data, then we can reopen the bug

Comment 11

[mikwit]

Hi
I have exactly same problem on 64bit Linux (Ubuntu) machine which is ONLY dedicated to archiving e-mails in my company (TB 60.8.0) and is equipped with SSD disk because it is handling large amount of e-mails.
Indexing is OFF and messages are kept in MBOX files (file per folder)

I marked 5068 messages in my SPAM folder and pressed DELETE at 10:31am which caused "moving to trash". It is 3:54pm right now (5h 23m later) and only 4316 emails has been processed - according to status bar.
All this time TB process takes 100% of 1 CPU core and 200~500MB or RAM.
Trash folder already contains 200000 emails, but still it takes way too long.
Before this operation I moved 8515 messages from Inbox to "SPAM training" folder where I keep my bayes training spam e-mails - containing approx 10000 messages before this operation. Moving messages took approx 1 minute - maybe less - I did not measure time (there was no reason).

I think time of operation depend on how many e-mails are in target folder.
IS there a way to check which operation takes that much time? Writing to file or updating indexes or so? Indexing is disabled in preferences. Is there any other indexing which is still working and can be OFF?

Regards
mikwit