991658 - unresponsive for > 3 minutes to delete 5000 EMails via IMAP, caused by copy 5000 mails and store flag \Deleted to 5000 mails. Only when using Trash folder - Delete immediate and mark as deleted work fine

Status: UNCONFIRMED

(MailNews Core :: Networking: IMAP, defect)

Description

[Harald Dunkel]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0 (Beta/Release)
Build ID: 20140208141556

Steps to reproduce:

To clean up I selected >5000 EMails within a single folder and hit [Delete]. (dedicated dovecot server in my LAN, using imap)



Actual results:

TB went guru for more than 3 minutes, not refreshing the screen or accepting any user input.


Expected results:

For reference I tried the same delete with another MUA, using a backup of my EMails: Claws takes only 5 seconds for the same delete operation.

Comment 1

[Wayne Mery (:wsmwk)]

profile on *local* disk?   Or external?

Did it take a long time selecting the messages?  Or only after hitting delete?

Comment 2

[WADA:World Anti-bad-Duping Agency]

(In reply to Harald Dunkel from comment #0)
> Steps to reproduce:
> To clean up I selected >5000 EMails within a single folder and hit [Delete].
> (dedicated dovecot server in my LAN, using imap)

Many already known issues are involved in your case.
(1) selected >5000 EMails => issue in selecting many mails
(2) issue in display of information about selected >5000 mails in message pane
    == issue around mail.operate_on_msgs_in_collapsed_threads = true
(3) hit [Delete] with selecting >5000 EMails
    => issue in generating a:b,c:d,...,y:z part in imap command like "uid a;b,c:d,...,y:z copy Trash"
    => issue after "uid a;b,c:d,...,y:z store Flags(\Deleted)".
       == issue around removing many MsgDBHdrs(==Similar to Shift+Delete at local mail folder)
          - issue around preparation of Undelete
          - issue around threading data update after delete of many mails
In any performane problem in above steps, "CPU 100% for a while", "unresponsive for a while" etc. can occur.

> Actual results:
. TB went guru for more than 3 minutes

Can you state detail of the "more than 3 minutes"?
  freeze time of Tb in each step.
Please keep "one bug per a problem" :-) (Who can do it in problem like above...)
Is problem other than issues listed in above steps involved in your case?
Which issue is irrelevant to your case?

Do you see your problem with following?
   IMAP delete model = Just mark it as deleted (Server Settings, When I delete a message:)
In this case, only "store flag \Deleted" is requested to server, and MggDBHdr is not removed. So, many steps in above is skipped.

Comment 3

[Harald Dunkel]

The .thunderbird directory is local. No remote file system is involved.

Selecting the EMails was as fast as usual. The thread view was active, all threads were expanded (using [*]) and marked (using [ctrl-a]). TB got stuck after hitting [Delete]. 

Server setting for deleting messages is: "Move it to Trash". If I set it to "mark as deleted", then TB doesn't get stuck, but all messages are just crossed out in the GUI. If I set it to "delete immediately", then the messages are not shown anymore. and TB doesn't get stuck, either.

Unfortunately TB seems to remember internally which messages got deleted. After the last test with "delete immediately" TB doesn't show the old EMails (restored from my backup) anymore, i.e. I have lost the test case :-(. Restarting dovecot or removing global-messages-db.sqlite don't help.

Hope this helps
Harri

Comment 4

[WADA:World Anti-bad-Duping Agency]

Difference between Just mark is as deleted and Remove immediately :
  After "uid xx store flgas(\Deleted),
    Just mark is as deleted : Keep MsgDBHdr with holding \Deleted status
                              If View=All,       show with strike-thru line, trash can icon
                              If View=Undeleted, hide at thread pane
    Remove immediately      : Remove MsgDBHdr, mail is not shown even with View=All
You can repeat test even after you used Remove immediately.
  Go "Just mark is as deleted" mode, then do Undelete mails with strike-thru line,
  and go back to "Remove immediately" mode.

Do you see your problem with following?
  IMAP Delete model : move to trash
  "Delete operation : Shift+Delete key with selecting 5000 mails
     Shift+Delete = delete without copy to Trash folder.
                    this is Undeletable.
     i.e. almost same as "Remove immediately"

Additional steps in Move to Trash, non Shift+Delete, is;
  (a) Preparation work to make "Undelete" possible 
  (b) execution of "uid a:b,c:d,...,y:z copy Trash"
In (b), Tb simply waits for OK response from server after Tb passed "uid ... copy Trash" command.
How long does it usually take to do "Copy 5000 mails from your IMAP Mbox to other IMAP Mbox"?

Comment 5

[WADA:World Anti-bad-Duping Agency]

Other concern at "move source folder".
  If move extention is supported by your Dovecot, Tb uses "move" instead of "copy + store \Deleted"
  when Delete with move to trash model and when manual/filter mail move.
  "Move" may be "copy + store \Deleted + expunge" at server.
  If so, and if IDLE is enabled(default=enabled), notification of "EXPUNGED" is sent from server
  immediately after end of "move" command.
Is IDLE enabled? Does yoy server support "Move" extension? EXPUNGED immeiately notified to Tb?

Difference between "Claws takes only 5 seconds for the same delete operation.".
If max cached connections is greater than 1(default is 5), Inbox and Trash is always selected at cached connection in Tb. In this case, Tb may immediately fetch "5000 copied mails in Trash" just after OK response to "uid '5000 mails' copy Trash". Because copied 5000 mails are absolutely new message in Trash folder, it same as "newly arrived mail to Inbox".
"fetch of copied 5000 mails to Trash" is almost same as "Repair Folder of Trash wit 5000 mails in Trash" in Tb.

How long does it usually take to do "Repair Folder of IMAP folder which holds 5000 mails in it" in yur environment?

Comment 6

[Harald Dunkel]

(In reply to WADA from comment #4)
> Do you see your problem with following?
>   IMAP Delete model : move to trash
>   "Delete operation : Shift+Delete key with selecting 5000 mails
>      Shift+Delete = delete without copy to Trash folder.
>                     this is Undeletable.
>      i.e. almost same as "Remove immediately"
> 

No, this took just a few seconds, after clicking on [continue] in the confirmation popup.

> Additional steps in Move to Trash, non Shift+Delete, is;
>   (a) Preparation work to make "Undelete" possible 
>   (b) execution of "uid a:b,c:d,...,y:z copy Trash"
> In (b), Tb simply waits for OK response from server after Tb passed "uid ...
> copy Trash" command.
> How long does it usually take to do "Copy 5000 mails from your IMAP Mbox to
> other IMAP Mbox"?

Depends. On the first run it took just about 5 to 10 seconds. I used Shift-Delete to remove the EMails in the target folder and tried again. On the second run I had to click several times on the [continue] button in the "unresponsive script" dialog. This took much more than 3 minutes.

Comment 7

[Harald Dunkel]

(In reply to WADA from comment #5)
> Is IDLE enabled? Does yoy server support "Move" extension? EXPUNGED
> immeiately notified to Tb?
> 
My Dovecot reports these capabilities:

IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED AUTH=GSSAPI

AFAICS the sources of the installed Dovecot explicitly mention support for RFC 6851 (the Move extension), but I do not know whether its active or if it should be reported as a capability as well.

How can I check for the EXPUNGED notification?

> Difference between "Claws takes only 5 seconds for the same delete
> operation.".
> If max cached connections is greater than 1(default is 5), Inbox and Trash
> is always selected at cached connection in Tb.

I am not sure if this is a good idea. I am sorting my EMails using procmail as a MDA, giving me a lot of folders with new EMails. Only a few EMails ignored by procmail (less than 1%) are stored in the INBOX.

Is there a cheat code in about:config to turn off this feature?

> How long does it usually take to do "Repair Folder of IMAP folder which
> holds 5000 mails in it" in yur environment?

About 5 seconds.

Hope this helps

Comment 8

[Harald Dunkel]

PS: My TB has "use IDLE if server supports it" set.

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to Harald Dunkel from comment #7)
> My Dovecot reports these capabilities:
> IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS
> LOGINDISABLED AUTH=GSSAPI
How IMAP clients can login wih LOGINDISABLED... :-)
> How can I check for the EXPUNGED notification?
It's unsolicit response from server to IDLE.
To see it, following is needed : At Tb-1 with IDLE enabled with NSPR logging : open Mbox, By other IMAP client including other Tb instance : expunge at Mbox(if Tb, delete at least one mail in Mbox, do Compact of the Mbox. Tb issues EXPUNGE).

IIRC, final/full CAPABILITY response is sent after login if Dovecot with ordinal setup.
I think Tb side NSPR logging is best for seeing IMAP level flow.
See bug 402793 comment #28. Because mail data is also logged, log data is usually huge. Get data with minimum configuration(no new mail check, click of smallest folder, smallest mail only, ..., or new profile with one accout only is defined), please.

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to Harald Dunkel from comment #6)
> > How long does it usually take to do "Copy 5000 mails from your IMAP Mbox to
> > other IMAP Mbox"?
> Depends. On the first run it took just about 5 to 10 seconds. I used
> Shift-Delete to remove the EMails in the target folder and tried again. On
> the second run I had to click several times on the [continue] button in the
> "unresponsive script" dialog. This took much more than 3 minutes.

At IMAP level, "what was done" is following only:
 1. With selecting SourceFolder at a cached connection, uid 5000_mails copy TargetFolder
 2. OK response from server
 3. With selecting TargetFolder at other cached connection, uid 5000_mails store Flags(\Deleted)
 4. OK response from server
 5. With selecting SourceFolder at a cached connection, uid 5000_mails copy TargetFolder
Other possible events:
- Download of entire mail data of all copied mails at Target folder after copy 5000 mails,
  if auto-sync is enabled and if TargetFolder is Offline-Use=On folder.
- Idexing by Global Search and Idexer, if copied mails are downloaded to offline-store by auto-sync. 
- Auto-Compact of Target Folder after delete 5000 mails
             = send EXUNGE to server
              + remove mail data for deleted mails from offline-store(file named Mbox, not Mbox.msf)
               (==if Offline-use=On, physical copy of data of all non-deleted mails to other file)

Anyway, step 1 + step 3 = 5 to 10 seconds. It seems for me performance issue is in "after copy 5000 mails" and "after store flag \Deleted to 5000 mails".
If "Delete mails" with move to trash model, "delete 5000 mails" occurs at SourceFolder and "new 5000 mails by copy" occurs at Trash.