720158 - Thunderbird hang copying many messages over IMAP, while moving many mails from IMAP folder to IMAP folder of different server by Drag&Drop of mails

Status: RESOLVED WORKSFORME

(Thunderbird :: Folder and Message Lists, defect)

Description

[Hubert Figuiere [:hub]]

Attachment: sample

I was copying 593 messages over IMAP from a remote server to a local (to the network) Courier-IMAP.

Thunderbird hang at 225 of 593.

Attaching a "sample"

Comment 1

[Hubert Figuiere [:hub]]

Attachment: second sample

this is the second sample a few minutes later.

Comment 2

[Hubert Figuiere [:hub]]

I use Thunderbird 10 from the beta channel (updated this morning) on MacOS X 10.6.x

Comment 3

[Hubert Figuiere [:hub]]

The remote server is Dreamhost. (FWIW).

Comment 4

[Hubert Figuiere [:hub]]

It might have been that the folder hadn't finished loading / synching locally. As when I restarted Thunderbird it was resynching said folder from the remote IMAP.

Comment 5

[WADA:World Anti-bad-Duping Agency]

> Bug summary: Thunderbird hang copying over IMAP

What phenomenon do you call by "hang"?
When status bar update stopped at "copied 225 of 593", Tb became unresponsive to any action? Or simply Tb stopped copy of 593 mails after copy 225 mails at Tb window(or a tab) where copy is requested?.
How did you requeste copy of 593 mails? Copy To? Drag&Drop of mails?
When copy stopped at 225 mails, was selection of 593 mails(highlight of mails) cleared?
How many mails was actually copied to local mail folder from IMAP folder?

Bug 538375 is for "bulk move of many mails from local folder to IMAP folder(opposite to your case) stops at mid of bulk move". In this case, it doesn't look network error happened while moving(uploading to IMAP server). 
Bug 716461 is for test result by emulation of "IMAP connection error while uploading" by pulling off of LAN cable.
"stop copying at mid of bulk copy" is seen in these bug too. 

Is external symptom of your case similar one to these bugs?
 
(In reply to Hub Figuiere [:hub] from comment #4)
> It might have been that the folder hadn't finished loading / synching
> locally. As when I restarted Thunderbird it was resynching said folder from
> the remote IMAP.

IIRC, bulk copy(bulk move too) from IMAP folder to local folder doesn't copy mail data in offline-store file. Bulk copy/move fetches mail data from server and write it to local mail folder. If new mail check, auto-sync, msgPurge etc. of this folder is invoked while copying multiple mails, it may interfere bulk mail copy.
Can you get NSPR log for imap, msgcopyservice etc.? (see bug 402793 comment #28)
> Win example : SET NSPR_LOG_MODULES=timestamp,imap:5,MsgCopyService:5,IMAPOFFLINE:5
Because many data is logged, get minimized log y next, please.
(1-1) stop automatic new mail check of any folder, set offline-use=off of as many folders as possible(if possible) to stop auto-sync.
(1-2) Restart Tb with logging enabled, and execute sae "copy 593 mails from IMAP folder to local folder".
(1-3) Problem is reproduced?
(2-1) If not reproduced, enable automatic new mail check(enabled for Inbox), and enable automatic new mail check of the copy source IMAP folder, se offline-use=on for the copy source IMAP folder.
(2-2) Restart Tb with logging enabled, and execute sae "copy 593 mails from IMAP folder to local folder".
(2-3) Problem is reproduced?
  
In any case, as seen in Bug 538375, log file becomes huge, if activity on irrelevant IMAP folders, unwanted activities occurs while getting log. Get NSPR log with "copy of small number of mails" for short while first, and check log file content, and reduce logs for irrelevant IMAP folders as many as possible by setting change, please. Even after it, IMAP log is large, because mail data is also logged. imap:4 may be a way to avoid mail data log. Please try it.

Comment 6

[Wayne Mery (:wsmwk)]

@hub does this match to bug 296453 for you?

unfortunately a trace without thunderbird/mozilla symbols isn't too helpful

Comment 7

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #6)
> @hub does this match to bug 296453 for you?

FYI.
That bug and friends of that bug are for problem in mass delete and mass delete step of bulk move.
This bug is for problem during bulk copy.
Bug 538375 / Bug 716461 refers to move too, but error occurs in copy step of bulk move if move case in these bug, so bug 296453, bug 538378 is currently irrevant.

Comment 8

[Hubert Figuiere [:hub]]

(In reply to WADA from comment #5)
> > Bug summary: Thunderbird hang copying over IMAP
> 
> What phenomenon do you call by "hang"?
> When status bar update stopped at "copied 225 of 593", Tb became
> unresponsive to any action? Or simply Tb stopped copy of 593 mails after
> copy 225 mails at Tb window(or a tab) where copy is requested?.


It was unresponsive. "Force Quit" was the only resolution.

> How did you requeste copy of 593 mails? Copy To? Drag&Drop of mails?

Select All, drag and drop.

> When copy stopped at 225 mails, was selection of 593 mails(highlight of
> mails) cleared?
> How many mails was actually copied to local mail folder from IMAP folder?

Not local mail folder. IMAP folder on the local network.... (the destination server is in my closet)
225 messages (maybe a few more) were copied.

Comment 9

[Hubert Figuiere [:hub]]

(In reply to Wayne Mery (:wsmwk) from comment #6)
> @hub does this match to bug 296453 for you?

Not really
 
> unfortunately a trace without thunderbird/mozilla symbols isn't too helpful

Unfortunately this is a recurring problem where it is hard to actually get the symbols, etc.

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to Hub Figuiere [:hub] from comment #8)
> Not local mail folder. IMAP folder on the local network....

Oh, sorry, I missed "network) Courier-IMAP" in next line after "local".
Bulk copy from IMAP folder to IMAP folder of different server case, so something wrong can happen on both "fetch from IMAP server" and "append to different IMAP server", and it can happen at same time.

> (In reply to WADA from comment #5)
> > > Bug summary: Thunderbird hang copying over IMAP
> > What phenomenon do you call by "hang"?
> It was unresponsive. "Force Quit" was the only resolution.

> Select All, drag and drop.

"Bulk move" case. Ctrl+Drag&Drop of mail == Copy, Drag&Drop of mail == Move.
(Bug 713814 exists, so Ctrl+Drag&Drop becomes Move, but it's Tb 9 only problem)
(Bug 713814 : Ctrl/Control+Drag moves a message instead of copying it)

If problem in "mass delete step", I think all 593 mails should be copied to local Courier-IMAP server because append is used(copy to different IMAP server) for each mail, and NNN of "copied NNN of 593" should be near to 593, although it can be problem in "mass delete step" in next case.
  MULTIAPPEND is used, and something wrong happened in local Courier-IMAP server
  after server returned OK response to MULTIAPPEND.
So I believe problem in "copy step of bulk move" instead of problem in "delete step of bulk move".

Questions to surely rule out problem in "mass delete step of bulk move" which is relevant to bug 296453 and his friends.
Drag&Drop of mails(move)? Ctrl+Drag&Drop of mails(copy)?
MULTIAPPEND is supportd by your Courier-IMAP server and is used by Tb? (check IMAP log, please)
When status bar update stopped at "copied 225 of 593" and Tb became unresponsive, was phenomenon like large Virtual and Real memory usage by Tb and/or CPU hog observed?

Can similar phenomenon be observed by "Pull off LAN cable while copying"? (emulation of network error while accessing IMAP server, like bug 716461)
(1) Copy same 593 mails
   (execute Copy instead of Move please, to surely rule out delete related issue)
(2) When some mails are copied, pull off LAN cable.
(3) Check Tb's status if possible. File/Offline/Work Offline. Checked status?
    Observe status bar, thread pane display.
(4) Open new Tb Window if possible(at account's context menu, Open).
    Check number of mails in copy target folder.
(5) Plug in LAN cable again. Tb perhaps goes back to Online mode automatically.

Comment 11

[WADA:World Anti-bad-Duping Agency]

Because "Move from IMAP folder to IMAP folder on different server" and it is executed while any other activity, such as new mail check, new mail notification via IDLE, auto-sync, msgPurge, auto-compact, can occur at same time, many issues may be relevant.
Setting dependency to bugs which is related to IMAP/copy/append/bulk mail copy/bulk mail move, for ease of tracking and problem analysis.

Comment 12

[WADA:World Anti-bad-Duping Agency]

Putting mass delete related bug in See Also because "bulk mail move" case, for ease of tracking.

Comment 13

[WADA:World Anti-bad-Duping Agency]

FYI.
Checked Tb's behaviour in "copy step of bulk mail move" when offline-use=on with Tb 9.0.1.
- Copy source folder : Gmail  IMAP folder, offline-use=on, already auto-sync'ed 
- Copy target folder : Yahoo! IMAP folder, offline-use=on, no mail in it
1. Select 1000 mails at Gmail IMAP folder, 2. Drag&Drop to Yahoo! IMAP folder, 3. When 10 mails are copied, Pull off Lan cable.
In IMAP log, there was no log for fetch from Gmail IMAP. Log for append to Yahoo! IMAP folder only.
So, as for "Copy step of bulk mail move + Pull off LAN cable while copying", it can be said that there is no difference from "Copy from local mail folder to IMAP folder + Pull off LAN cable while copying"(== Bug 716461), except that Copy source folder is affected by Offline mode due to "Pull off LAN cable" when Gmail IMAP folder case.
In above test, selection of mails at "Gmail IMAP folder(move is rquested at here) was cleared after "Pull off LAN cable"(Tb goes Offline mode). This is perhaps a result of "entering Work Offline mode" and is a difference from actual local mail folder case.

Comment 14

[WADA:World Anti-bad-Duping Agency]

Quick summary of this bug.

This bug is for following case.
- Move many mails by Drag&Drop of selected messages
- Move between an IMAP folder-A on IMAP server-A to IMAP folder-B on other IMAP server-B
- "hang" in bug summary == "unresponsive". "Force Quit" was the only resolution.
- The "unresponsive" looks to happen while "copying".
  i.e. While (a) "fetch from Folder-A on IMAP-A" and (b) "append to Folder-B on IMAP-B".
- CPU usage, Memory usage or increase while "unresponsive" is not reported.

If Offline-Use=On, and if auto-sync is active, auto-sync may interfere (i) and (ii).
But "Offline-Use=On/Off" nor "auto-sync=enabled/disabled" is not provided yet.
If MULTIAPPEND is supported by server and if Tb utilized MULTIAPPEND in (ii), it may affect on problem.
But "MULTIAPPEND=supported/not-supported" is not provided yet.

Because IMAP process is done at main task and IMAP access is synchronous, if N times of ("fetch of many mails from Folder-A of IMAP-A" + "append of many mails to Folder-B of IMAP-B") is executed for an "Move many many mails between an IMAP folder-A on IMAP server-A to IMAP folder-B on other IMAP server-B", "unresponsive for a while" itself is ordinal phenomenon.
It's by current design. It can be called as "restriction in current Tb".
However, "what happens if wait for long time sufficiently" is not provided yet.

If problem depends on "number of mails", data like next is usually needed.
  Move     10 mails : OK. Took nn_10  seconds
  Move    100 mails : OK. Took nn_100 seconds
  Move   1000 mails : OK. Took nn_1000 minutes
                                      <= "Time to take move" is O( number of mails )?
                                         "Time to take move" is O( number of mails **2 )?
                                      <= Something bad started to occur
  Move  10000 mails : BAD. Killed after MM minute
  Move  20000 mails : BAD  Killed after MM minute
This kind of data is not available yet.

Comment 17

[crosbow]

oh nice, 5 years later still not fixed ^^

i tried to move 900 emails into a different IMAP folder on the same server and firefox freezes instantly. after some time it asks if i want to continue the script or stop it.

chrome://messenger/content/folderPane.js:2031

after restarting the dialog asking to stop the script didnt appear and firefox was just frozen.

Comment 18

[gene smith]

(In reply to crosbow from comment #17)

oh nice, 5 years later still not fixed ^^

You mean tb, not ff, right? Been testing doing same thing with >7000 emails in folder and don't see what you describe.

Comment 19

[Wayne Mery (:wsmwk)]

Does this still reproduce for you when using version 102 ?

Comment 20

[kontakt]

Problem was solved, sorry. The reason was the transferred emails were copied/migrated over to different servers a lot of times. Due to that reason some mail client in between "appended all mails" together somehow. So file 1 had one email inside, the second file the first one and the second one, the third file all 3 files together. And that went up to 30000.

This faulty files made Thunderbird hang up when trying to move them. The way I fixed it: I exported the mails (even as big as they were) and then imported them into Roundcube to my IMAP server. In the process Roundcube cleaned the files and Thunderbird was running fine again.

Weird process, but at least I got still all my mails from the last 20 years.

Regards,
Dennis

Comment 21

[kontakt]

(In reply to kontakt from comment #20)

Problem was solved, sorry. The reason was the transferred emails were copied/migrated over to different servers a lot of times. Due to that reason some mail client in between "appended all mails" together somehow. So file 1 had one email inside, the second file the first one and the second one, the third file all 3 files together. And that went up to 30000.

This faulty files made Thunderbird hang up when trying to move them. The way I fixed it: I exported the mails (even as big as they were) and then imported them into Roundcube to my IMAP server. In the process Roundcube cleaned the files and Thunderbird was running fine again.

Weird process, but at least I got still all my mails from the last 20 years.

Regards,
Dennis

(In my case at least, didnt see that this isnt even my Bug Thread)

Comment 22

[Hubert Figuiere [:hub]]

I haven't seen this in a long time.

Comment 23

[Wayne Mery (:wsmwk)]

Thanks for the updates folks.
Even if it still exists, is adequately covered in other bugs, for example bug 538375 and bug 716461

Comment 24

[gene smith]

Removing dep on bug 1828372. (Not sure why I marked it as dependent.)