538375 - When attempting to copy or move large numbers of local messages to an IMAP server, it fails without any error or status message. Related to timeouts

Status: NEW

(Thunderbird :: Folder and Message Lists, defect)

Description

[Magritte]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0

I've been attempting to migrate some local mail to Gmail through IMAP. I've been able to move many folders which have a relatively small number of messages (in the few hundreds) but for folders with many messages (eg. over 4000) the move or copy operation simply stops partway through. The number of messages copied will vary between a few dozen, hundred or even a couple of thousand but it will never successfully finish. There is no error message or warning at all, neither as a popup dialog or in the status bar. It just stops trying without any information provided. 

If I select a small number of messages from the folder I can successfully move them.

Furthermore, Thunderbird does not delete local messages that WERE successfully moved. It will only delete local messages if all messages were successfully copied. This makes it impossible to simply keep trying until all the messages have finally been moved successfully, since the copied messages remain in the local folder and therefore just get copied again during the next attempt. Moving or copying all the messages appears to be impossible.

At fist I suspected there may be a Gmail-specific issue. However, I tried installing a local IMAP server (hMailServer) and verified that the same issue occurs even on the local IMAP server. Small numbers of messages may be transferred, but it always quits without warning when attempting to move large amounts of mail.

Reproducible: Always

Steps to Reproduce:
1.Select several thousand email messages (or a folder with several thousand messages)
2.Shift+Drag to IMAP server or right-click, select move-to, then select a folder on the IMAP server. (You may have to first create an empty folder on the IMAP server)
3.Wait as some of the messages are moved, but then it stops transferring the rest without any warning.
Actual Results:  
Only a subset of selected messages are successfully moved to the IMAP server and those messages are not deleted from the local folder.

Expected Results:  
All messages are transfered to the IMAP server and any successfully transfered message is deleted from the local folder (assuming you chose to move rather than copy).

Comment 1

[Wayne Mery (:wsmwk)]

Magritte, does Message|Move to|...   or r-click Move to|... work better?

Comment 2

[Magritte]

No, I've tried dragging, as well as right-click move to and message move to and they all behave the same. I've also tried using Nostalgy's hotkeys and they didn't work at all. When dragging I've tried dragging files to an existing folder, dragging the folder to the account root, and either simple or shift+drag (Thunderbird has a sad lack of appropriate feedback when dragging and dropping and it's not clear when dragging messages to a folder if it will be a copy or move operation). 

In any case, I just tried again using a local IMAP server and a Thunderbird profile with no extensions installed. Attempting to move a folder with around 9000 messages successfully moved about 4000 before stopping without any indication as to why. The Activity Manager window displays NO information about uploading mail, only downloading/syncing.

Again, because those emails that were uploaded during a move operation were not removed from the original folder, there's no simple way to determine which messages are missing or any easy way to continue from where the operation stopped. Messages should be deleted as they are uploaded, but Thunderbird appears to want to upload all messages first before deleting, and then if the upload is not completed you're left with no messages deleted locally and some subset copied to the server.

Comment 3

[Wayne Mery (:wsmwk)]

thanks for checking that.  is it likely/unlikely that a new message arrived while your message operations were in progression?

Comment 4

[Magritte]

When I tried this using a local IMAP server that would have been impossible. However, I have noticed that it will start syncing to the new messages in the IMAP folder while I'm trying to move my messages into it. So it starts copying then it starts downloading from the server to the local IMAP store, then continues copying, etc. Sometimes the message in the status bar alternates quickly from downloading to copying. It's never been clear to me if uploading and downloading occur simultaneously or not?

Maybe I'll try disabling automatic mail check and IDLE support to see if I can prevent it from checking for new mail while uploading.

Comment 5

[Magritte]

I tried disabling the following server settings:
Check for new messages at startup
Check for new messages every ___ minutes
Use IDLE command if the server supports it
Keep messages for this account on this computer

At first I thought this might work as it got further than ever before: around 8000 of 9000 messages from one folder. However, at that point, it stopped like all the other times and since then no improvement on subsequent attempts even after closing and restarting Thunderbird.

What's worse is at one point I accidentally moved the messages from the local folder to the IMAP folder while Thunderbird was in offline mode. This caused all the messages to be deleted from the local folder and appear in the IMAP folder. However, once I went back to Online mode, it started copying those messages to the IMAP server, and then stopped about half way through. As a result, half the messages never appeared on the IMAP server but did disappear from the local folder.

After restarting Thunderbird it did appear to try again to upload those messages (so I assume they are still cached somewhere) but it again quite after only a couple of hundred messages.

At this point I'm using a local IMAP server for speed and working with a backup of the actual profiles. However, at least in some cases there does appear to be a potential for serious data loss due to this bug if a large move is attempted while in offline mode.

Comment 6

[Pehr Söderman]

Looks like a duplicate of 381867

Comment 7

[WADA:World Anti-bad-Duping Agency]

Magritte(bug opener), can you please get IMAP log with timestamp and check IMAP level flow, and check at which step "copy of mails to IMAP folder" stops or is interfered.
> https://wiki.mozilla.org/MailNews:Logging
> http://www.mozilla.org/projects/nspr/reference/html/prlog.html#25328
> SET NSPR_LOG_MODULES=timestamp.imap:5
If Tb's fault is seen in log, attach IMAP log file, if all data in log can be opened to public, please.

Comment 8

[Magritte]

I finally got around to making a log. I'm not comfortable uploading it as it's both rather large and contains all the text of many e-mails. I tried to look through it to find the point of failure but couldn't really identify what it was. I tried to upload a folder to a Gmail account. It doesn't help that it tries to download newly uploaded messages. Actually, the first time I tried it synced a mess of messages from the server and the log was over 4 GB in size. Thinking that everything should have been synced up, I tried again. Even then the log is around 700 MB.

Is there anything in particular I can look/search for in the log?

One thing I notice is I select all the files in the folder for moving to the server. When it fails, the selection changes (without any interaction from me) from everything selected to a single message selected.

Comment 9

[Magritte]

(In reply to comment #6)
> Looks like a duplicate of 381867

It seems different. 381867 states that in the endless loop messages are copied multiple times to the server. I have not seen this behaviour. Each message is copied a single time to the server, but Thunderbird is unable to complete the copying if many messages are selected. I initially found this on Gmail, so in principle there could have been an endless loop at the last message copied since Gmail automatically removes duplicates messages. However, I subsequently confirmed the same behaviour using the open source hMailServer IMAP server running on the same local machine and did not find any instances of a single or multiple messages being copied more than once.

Comment 10

[Ludovic Hirlimann [:Usul]]

(In reply to comment #8)
> I finally got around to making a log. I'm not comfortable uploading it as it's
> both rather large and contains all the text of many e-mails. I tried to look

You can remove the message content. and can anonymize the log, this should make it smaller. If you are still uncomfortable about sharing it publicly you can send it to [e-mail] with a reference to the bug number. But trimming and cleaning the log from personal data and attaching it here is the preferred method.

Comment 11

[Wayne Mery (:wsmwk)]

fwiw my comment 3 refers to bug 458570

Comment 12

[Magritte]

So I wasn't comfortable uploading the log file with real messages and removing personal data didn't seem like a trivial thing to do. Finally I thought of a way to generate a large sample set of messages to use instead: subscribing to individual e-mail updates for a highly active google group. This generated over 5000 messages in about a day which I copied to a local folder then tried to upload (move) to a test gmail account over IMAP. It successfully copied 3099 of 5142 messages before it stopped. Again, there was no error message, all of the messages in the local folder remained highlighted, and none were deleted.

The log file was too large to attach to this bug report, so I've upload it here:
https://docs.google.com/leaf?id=0B4NCPWEUDyRaYzM3ZWY5OWYtZjQ2Zi00ZDM1LWI1MjktYWNkZGFlY2EyNDQz&hl=en

Thanks.

Comment 13

[Wayne Mery (:wsmwk)]

thanks  Magritte

Comment 14

[WADA:World Anti-bad-Duping Agency]

(In reply to Magritte from comment #12)
> It successfully copied 3099 of 5142 messages before it stopped.
> Again, there was no error message, all of the messages in the local folder
> remained highlighted, and none were deleted.

From where does the "3099 of 5142" come from?
Status bar message? Activity manager message? Actually selected & copied mail count?

How many mails were selected at move source folder(local mail folder), and how many mails were actually copied to move target folder?
(As Tb doesn't show total mail count, operation like "Select All", "Mark as Unread" is needed, unless add-on is installed)

If "Order Received" column(if IMAP, value is UID of mail) is shown, you can see UID of mail. 
What is UID of first copied mail? (smallest one in copied mails)
What is UID of last copied mail?  (largest one if additional mail is not added)
Because UID is incremented by one, UID of bulk copied mails are usually continuous, unless new mail arrives while you are uploading many mails.
When sorted by "Order Received" column, do you see non continuous UID?

Gmail IMAP actually returns response like next.
> http://www.google.com/support/forum/p/gmail/thread?tid=166a48b67e267726&hl=en#fid_166a48b67e267726000473da9b2938ae
> host:imap.gmail.com -- port:993 -- 2.47 NO [ALERT] Account exceeded bandwidth limits. (Failure)

No such message in Activity Manager panel?

Gmail doesn't permit duplicated mail data. So, if duplicated mail data is uploaded, Gmail finaly ignores it, although Gmail assigns UID for the dup'ed mail in the uploaded folder. This is same for Tb as "uploaded mail by Tb is deleted & expunged by some one just after Tb uploaded the mail(successful append command)".
Although this can't explain phenomenon of "Selection of all mails remains at move source folder, and mails are never deleted", it can explain some part of of "number of copied mails is smaller than number of mails for which move was requested".
Please surely rule out such case, please.

I uploaded(copied from local mail folder) 10,000 mails(1KB per a mail, 10MB total, never dup of existent mails, no dup in uploaded mails) to Gmail IMAP's newly created folder(starts from UID=1) today using Tb 9.0.1 on Win, for testing of other bug. It took long because of 10MB data, but it normally ended, and UID was continuous.
Bug opener, do you still see your problem with recent Tb release?

Comment 15

[WADA:World Anti-bad-Duping Agency]

Attachment: Extracted IMAP log (append command and response only)

(In reply to Magritte from comment #12)
> The log file was too large to attach to this bug report, (snip)
> https://docs.google.com/leaf?id=0B4NCPWEUDyRaYzM3ZWY5OWYtZjQ2Zi00ZDM1LWI1MjktYWNkZGFlY2EyNDQz&hl=en

Yes, your log file is too large. More than 800 MB!
More than half was fetch logs from [Gmail]/All Mail. Why not stop auto mail check etc. even though check with DLE enabled and with Gmail IMAP?

I extracted log line for "append command" and "append response" only, and merged command and response into single line for ease of analysis.
Example of extracted log line(log for last append command).
> 2010-01-10 19:18:22.692000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 7142 append "imapsamplemailbox" "09-Feb-2010 15:52:28 -0500" {12544+}
> "2010-01-10 19:18:23.503000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 7142 OK [APPENDUID 632907681 3179] (Success)
 
As seen in extracted/merged log, appended mail is assigned UID from 1 to 3179, and all append request is successfull. After it, log for fetch of uploaded maiis from imapsamplemailbox is seen, and all of UID= 1 to 3179 is also successfully fetched, and finaly IMAP session is normally closed because you terminated Tb.
So, "duplicated mail" is not involved in 3179 mails from UID=1 to UID=3179.

(In reply to Magritte from comment #12)
> It successfully copied 3099 of 5142 messages before it stopped.
> Again, there was no error message, all of the messages in the local folder
> remained highlighted, and none were deleted.

Perhaps "3099 of 5142" is status bar message.
Tb doesn't update status bar message upon each append. So smaller number than actually appended mails is shown.
An IMAP task of Tb for upload of mail is perhaps waiting for next request from task for "move mails".

How did you move mails?
Select All + Drag&Drop? or "Move To" context menu? 

IIRC, problem/bug like next existed.
  Move of may mails by Drag&Drop somehow stops at mid of move of may process.
It may be caused by new mail arrival during move of many mails, by other request(s) to same folder which happened at same time.
"Event of Drag&Drop disappers while processing due to other event(s)" like issue may be a cause.

Comment 16

[WADA:World Anti-bad-Duping Agency]

Per attached log data and your report, confirming.

Comment 17

[WADA:World Anti-bad-Duping Agency]

FYI.
Following are bugs for "move of mails by Drag&Drop" found by serch for "move drag" in bug summary.
> Bug 368112 Messages lost when multiple messages moved between local mail folders
>            (not only move/drag&drop but also copy/thru menu)
> Bug 452832 drag and drop multiple messages between folders fails (moves only one)
Both are already FIXED.
"Messages in local move source folder didn't disappear in your case" may be a effect by fix of Bug 368112(see many dups of that bug).

Comment 18

[WADA:World Anti-bad-Duping Agency]

Checked Tb's behavior when offline is forced by pull-off of LAN cable during move mails from local folder to IMAP folder", using Tb 9.0.1 in Win-XP. 

(1) At Tb Window-1, using "Move To", request move of 1000 mails in local mail folder to Gmail IMAP mail folder, Inbox/Inbox3.

> 3820] 620[4d87fc0]: 4a9a800:imap.gmail.com:S-INBOX/Inbox3:CreateNewLineFromSocket: 37 OK [APPENDUID 602313603 152] (Success) 
> [3820] 620[4d87fc0]: 4a9a800:imap.gmail.com:S-INBOX/Inbox3:SendData: 38 noop 
> [3820] 620[4d87fc0]: ReadNextLine [stream=54ff880 nb=15 needmore=0]
> [3820] 620[4d87fc0]: 4a9a800:imap.gmail.com:S-INBOX/Inbox3:CreateNewLineFromSocket: 38 OK Success 
> [3820] 0[100f140]: CopyNextStreamMessage: Copying 10 of 10000
> (End of upload of 10-th mail)

> (Start of upload of mail = 11-th mail)
> [3820] 620[4d87fc0]: 4a9a800:imap.gmail.com:S-INBOX/Inbox3:ProcessCurrentURL: entering
> [3820] 620[4d87fc0]: 4a9a800:imap.gmail.com:S-INBOX/Inbox3:ProcessCurrentURL:imap://[e-mail]:993/appendmsgfromfile%3E/INBOX/Inbox3:  = currentUrl
> [3820] 620[4d87fc0]: 4a9a800:imap.gmail.com:S-INBOX/Inbox3:SendData: 39 append "INBOX/Inbox3" (\Seen tag-10) "01-Jan-1970 09:09:00 +0900" {938} 
> [3820] 620[4d87fc0]: ReadNextLine [stream=54ff880 nb=12 needmore=0]
> [3820] 620[4d87fc0]: 4a9a800:imap.gmail.com:S-INBOX/Inbox3:CreateNewLineFromSocket: + go ahead 
> [3820] 620[4d87fc0]: 4a9a800:imap.gmail.com:S-INBOX/Inbox3:SendData: X-Mozilla-Keys: tag-10                                                                           
> (logs for mail body lines are omitted)
> (following is indicator of "End of message data", so Tb waits for OK response to append command)
> [3820] 620[4d87fc0]: 4a9a800:imap.gmail.com:S-INBOX/Inbox3:SendData:  

(2) Pull off LAN cable.

(After 2 seconds, next log was seen, because I used DebugView for logging data)
[MUP]: DfscFsctrlStateTransition invoked.
[MUP]: Flushing pkt cache...
[MUP]: done.

(3) Tb went Work Offline mode, because Tb detected offline status.

(4) At Tb Window-1, following was still shown at status bar,
      Copying Message 11 of 1000 to Inbox3
    and progress par was spinning, and selection of mails were kept.

(5) Plug in LAN cable.
    Tb went back to Work Online mode, because Tb detected online status.

(6) Open Tb Window-2, select Inbox/Inbox3 of Gmail IMAP account.
    Mail of UID=11 was normally shown, because all data was passed to server,
    although Tb couldn't receive OK response.

(7) At Tb Window-1, following was/is shown at status bar,
      Copying Message 11 of 1000 to Inbox3
    and progress bar didn't stop and is still spinning,
    even after "Repair Folder" at Tb window-2.
    Selection of mails were still kept.

When I executed above test first time, error of "unable to connect to imap.gmail.com" was shown at Activity Manaer panel, and spinning at Tb Window-1 didn't occur.
"spinning at Tb Window-1" is perhaps timing dependent phenomenon.
"silent stopping of move operation at middle of bulk move" is perhaps common phenomenon when silent(for Tb) connection loss occurs during bulk move/copy.

Above phenomenon is very similar to phenomenon you saw, although "connection loss actually happend in your case or not" is unknown.
If connection with IMAP server is silently lost, pheomenon similar to above can happen.
As as many untagged response of "* MMM EXPUNGE, * NNN EXISTS" where MMM=NNN or MMM=NNN-1 to append command is seen in your log, you uploaded many dup mails to Gmail IMAP server when you got IMAP log.
  * MMM EXPUNGE = MMM-th mail in existent mails is expunged.
  * NNN EXPUNGE = There is NNN mails, after above expunge.
  When Gmail IMAP above untagged response means;
    MMM-th mail(UID=X) is same as newly appended mail(UID=Y, larger than X),
    so mail of UID=10 is removed, and mail of UID=Y is added.
    MMM=NNN  : Mail of UID=X is mail appended just before this append command.  
    MMM=NNN-1: Mail of UID=X is mail appended by former append command.  
  This is because Gmail does't keep dup mails.
Abnormal condition like above may produces unwanted workload of server. So, silent(for Tb) connection kill by Gmail may happen.

Magritte(bug opener), did silent connection loss occur in your case?
Can you reproduce your problem with recent Tb release?

Comment 19

[WADA:World Anti-bad-Duping Agency]

Attachment: IMAP log : Pull off LAN cble while bulk move of mails from local folder to Gmail IMAP folder

(1) 5 mails exist in Gmail IMAP folder, Inbox/Inbox3
  UID=100, Subject: MANYMAILS-000011, Message-ID: <MANYMAILS.000011.000001>
  UID=102, Subject: MANYMAILS-000013, Message-ID: <MANYMAILS.000013.000001>
  UID=104, Subject: MANYMAILS-000015, Message-ID: <MANYMAILS.000015.000001>
  UID=106, Subject: MANYMAILS-000017, Message-ID: <MANYMAILS.000017.000001>
  UID=108, Subject: MANYMAILS-000019, Message-ID: <MANYMAILS.000019.000001>
(2) At local mail folder, request move 10000 mails to the Gmail IMAP folder,
    includeng same 5 mails as existent mails in Gmail IMAP Inbox/Inbox3.
    Because "move/copy from local to IMAP" is done from bottom mail to top mail
    in sorted order at thread pane, mails are sorted in reversed order of
    Subject: to force required append order.
(3) Because of Gmail IMAP, following is returned to append of dup mails.
> Message-ID: <MANYMAILS.000011.000001> => * 1 EXPUNGE, * 15 EXISTS (UID=100 is removed as dup)
> Message-ID: <MANYMAILS.000013.000001> => * 1 EXPUNGE, * 16 EXISTS (UID=102 is removed as dup)
> Message-ID: <MANYMAILS.000015.000001> => * 1 EXPUNGE, * 17 EXISTS (UID=104 is removed as dup)
> Message-ID: <MANYMAILS.000017.000001> => * 1 EXPUNGE, * 18 EXISTS (UID=106 is removed as dup)
> Message-ID: <MANYMAILS.000019.000001> => * 1 EXPUNGE, * 19 EXISTS (UID=108 is removed as dup)
(4) Pull off LAN cable.
Because connection error happend just after send of "noop" in this test, spining at Window-1(where move mails is requesed) was observed.
> 00001693	120.09008026	[5380] 3372[512cc40]: 5dcd000:imap.gmail.com:S-INBOX/Inbox3:SendData: 85 append "INBOX/Inbox3" (\Seen tag-1) {938} 	
> 00001697	120.26193237	[5380] Message-ID: <MANYMAILS.000031.000001> 	
> 00001701	120.26193237	[5380] Subject: MANYMAILS-000031 	
> 00001726	120.59330750	[5380] 3372[512cc40]: 5dcd000:imap.gmail.com:S-INBOX/Inbox3:CreateNewLineFromSocket: * 31 EXISTS 	
> 00001728	120.59339905	[5380] 3372[512cc40]: 5dcd000:imap.gmail.com:S-INBOX/Inbox3:CreateNewLineFromSocket: 85 OK [APPENDUID 602313603 193] (Success) 	
> 00001729	120.59358215	[5380] 3372[512cc40]: 5dcd000:imap.gmail.com:S-INBOX/Inbox3:SendData: 86 noop 	

==> LAN cable was pulled off at here.

> 00001730	121.76026917	[MUP]: DfscFsctrlStateTransition invoked.	
> 00001731	121.76027679	[MUP]: Flushing pkt cache...	
> 00001732	121.76029205	[MUP]: done.	

(5) Plug-in of LAN cable.
Tb went back Online mode automatically, but server access for Inbox/Inbox3 was impossible in this test. Nothing was requested to server for Inbox/Inbox3 via connection for Inbox/Inbox3, even by Repair Folder at newly opened Tb Window-2.
So, I terminated Tb and restarted Tb, to check.

> 00001810	733.12371826	[5380] 6040[5d7d5c0]: ImapThreadMainLoop leaving [this=13006000]
=> Terminate Tb, Restart Tb at here.
> 00001811	821.73992920	[2572] 1160[512cc40]: ImapThreadMainLoop entering [this=5dcd000]	
> 00001812	821.74041748	[2572] 0[100f140]: 5dcd000:imap.gmail.com:NA:SetupWithUrl: clearing IMAP_CONNECTION_IS_OPEN

Comment 20

[WADA:World Anti-bad-Duping Agency]

Attachment: Your IMAP log for imapsamplemailbox, after last append(UID=3179)

(1) Last append

> 2010-01-10 19:18:22.692000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:ProcessCurrentURL: entering
> 2010-01-10 19:18:22.692000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:ProcessCurrentURL:imap://[e-mail]:993/appendmsgfromfile%3E/imapsamplemailbox:  = currentUrl
> 2010-01-10 19:18:22.692000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 7142 append "imapsamplemailbox" "09-Feb-2010 15:52:28 -0500" {12544+}
> 2010-01-10 19:18:23.503000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * 3099 EXISTS
> 2010-01-10 19:18:23.503000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 7142 OK [APPENDUID 632907681 3179] (Success)
> 2010-01-10 19:18:23.503000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 7143 noop
> 2010-01-10 19:18:23.690000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 7143 OK Success

(2) IDLE afterlast appnd
> 2010-01-10 19:18:23.721000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 7144 IDLE
> 2010-01-10 19:18:23.893000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: + idling

(3) After 5 minutes of idling, unsol "* 3099 EXISTS", and Tb terminates IDLE 
> 2010-01-10 19:23:29.357000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * 3099 EXISTS
> 2010-01-10 19:23:29.357000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: DONE
> 2010-01-10 19:23:29.529000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 7144 OK IDLE terminated (Success)

(4) Then Tb request "check", and fetch flag of newly appended mails
> 2010-01-10 19:23:29.529000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:ProcessCurrentURL: entering
> 2010-01-10 19:23:29.529000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:ProcessCurrentURL:imap://[e-mail]:993/select%3E/imapsamplemailbox:  = currentUrl
> 2010-01-10 19:23:29.529000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 7145 check
> 2010-01-10 19:23:29.700000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 7145 OK Success
> 2010-01-10 19:23:29.716000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 7146 getquotaroot "imapsamplemailbox"
> 2010-01-10 19:23:29.888000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 7146 OK Success
> 2010-01-10 19:23:29.888000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 7147 UID fetch 671:* (FLAGS)
> 2010-01-10 19:23:30.137000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: *  654 FETCH (UID 671 FLAGS (NonJunk))
>                                                                                          (untagged responses is omitted)     |
> 2010-01-10 19:23:30.824000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * 3099 FETCH (UID 3179 FLAGS ())
> 2010-01-10 19:23:30.824000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 7147 OK Success

(5) And fetch headers of mails
> 2010-01-10 19:23:30.839000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 7148 UID fetch 671:3179 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type)])
> 2010-01-10 19:23:31.104000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * 654 FETCH (UID 671 RFC822.SIZE 12412 FLAGS (NonJunk) BODY[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type)] {321}
> 2010-01-10 19:23:31.104000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:STREAM:OPEN Size: 12412: Begin Message Download Stream
> 2010-01-10 19:23:31.120000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:STREAM:CLOSE: Normal Message End Download Stream
>                                                                         (logs after fetch of UID=671, before fetch of UID=3179, are omitted)
> 2010-01-10 19:23:38.093000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * 3099 FETCH (UID 3179 RFC822.SIZE 12544 FLAGS (NonJunk) BODY[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type)] {348}
> 2010-01-10 19:23:38.093000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:STREAM:OPEN Size: 12544: Begin Message Download Stream
> 2010-01-10 19:23:38.093000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:STREAM:CLOSE: Normal Message End Download Stream
> 2010-01-10 19:23:38.093000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 7148 OK Success

(6) fetch of message body is invoked
> 2010-01-10 19:23:38.140000 UTC - 0[a2d140]: queuing url:imap://[e-mail]@imap.googlemail.com:993/fetch>UID>/imapsamplemailbox>671,673:674
> 2010-01-10 19:23:38.140000 UTC - 0[a2d140]: considering playing queued url:imap://[e-mail]@imap.googlemail.com:993/fetch>UID>/imapsamplemailbox>671,673:674
> 2010-01-10 19:23:38.140000 UTC - 0[a2d140]: creating protocol instance to play queued url:imap://[e-mail]@imap.googlemail.com:993/fetch>UID>/imapsamplemailbox>671,673:674
> 2010-01-10 19:23:38.140000 UTC - 0[a2d140]: failed creating protocol instance to play queued url:imap://[e-mail]@imap.googlemail.com:993/fetch>UID>/imapsamplemailbox>671,673:674
> 2010-01-10 19:23:38.156000 UTC - 0[a2d140]: considering playing queued url:imap://[e-mail]@imap.googlemail.com:993/fetch>UID>/imapsamplemailbox>671,673:674
> 2010-01-10 19:23:38.156000 UTC - 0[a2d140]: creating protocol instance to play queued url:imap://[e-mail]@imap.googlemail.com:993/fetch>UID>/imapsamplemailbox>671,673:674
> 2010-01-10 19:23:38.156000 UTC - 0[a2d140]: playing queued url:imap://[e-mail]@imap.googlemail.com:993/fetch>UID>/imapsamplemailbox>671,673:674
> 2010-01-10 19:23:38.156000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:ProcessCurrentURL: entering
> 2010-01-10 19:23:38.156000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:ProcessCurrentURL:imap://[e-mail]:993/fetch%3EUID%3E/imapsamplemailbox%3E671,673:674:  = currentUrl
> 2010-01-10 19:23:38.156000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 7149 UID fetch 671,673:674 (UID RFC822.SIZE BODY.PEEK[])
> 2010-01-10 19:23:38.390000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * 654 FETCH (UID 671 RFC822.SIZE 12412 BODY[] {12412}
> 2010-01-10 19:23:38.390000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:STREAM:OPEN Size: 12412: Begin Message Download Stream
> 2010-01-10 19:23:38.421000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:STREAM:CLOSE: Normal Message End Download Stream
> 2010-01-10 19:23:38.452000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * 655 FETCH (UID 673 RFC822.SIZE 12587 BODY[] {12587}
> 2010-01-10 19:23:38.452000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:STREAM:OPEN Size: 12587: Begin Message Download Stream
> 2010-01-10 19:23:38.452000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:STREAM:CLOSE: Normal Message End Download Stream
> 2010-01-10 19:23:38.577000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * 656 FETCH (UID 674 RFC822.SIZE 11998 BODY[] {11998}
> 2010-01-10 19:23:38.577000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:STREAM:OPEN Size: 11998: Begin Message Download Stream
> 2010-01-10 19:23:38.608000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:STREAM:CLOSE: Normal Message End Download Stream
> 2010-01-10 19:23:38.608000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 7149 OK Success

(7) "fetch of message body" continues. It looks 2 or 5 mails arerequested by each fetch.

(8) Last continuous "fetch of message body"
> 2010-01-10 19:30:43.810000 UTC - 0[a2d140]: queuing url:imap://[e-mail]@imap.googlemail.com:993/fetch>UID>/imapsamplemailbox>2396
> 2010-01-10 19:30:43.810000 UTC - 0[a2d140]: considering playing queued url:imap://[e-mail]@imap.googlemail.com:993/fetch>UID>/imapsamplemailbox>2396
> 2010-01-10 19:30:43.810000 UTC - 0[a2d140]: creating protocol instance to play queued url:imap://[e-mail]@imap.googlemail.com:993/fetch>UID>/imapsamplemailbox>2396
> 2010-01-10 19:30:43.811000 UTC - 0[a2d140]: failed creating protocol instance to play queued url:imap://[e-mail]@imap.googlemail.com:993/fetch>UID>/imapsamplemailbox>2396
> 2010-01-10 19:30:43.812000 UTC - 0[a2d140]: considering playing queued url:imap://[e-mail]@imap.googlemail.com:993/fetch>UID>/imapsamplemailbox>2396
> 2010-01-10 19:30:43.812000 UTC - 0[a2d140]: creating protocol instance to play queued url:imap://[e-mail]@imap.googlemail.com:993/fetch>UID>/imapsamplemailbox>2396
> 2010-01-10 19:30:43.812000 UTC - 0[a2d140]: playing queued url:imap://[e-mail]@imap.googlemail.com:993/fetch>UID>/imapsamplemailbox>2396
> 2010-01-10 19:30:43.812000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:ProcessCurrentURL: entering
> 2010-01-10 19:30:43.812000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:ProcessCurrentURL:imap://[e-mail]:993/fetch%3EUID%3E/imapsamplemailbox%3E2396:  = currentUrl
> 2010-01-10 19:30:43.812000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 8230 UID fetch 2396 (UID RFC822.SIZE BODY.PEEK[])
> 2010-01-10 19:30:45.868000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * 2331 FETCH (UID 2396 RFC822.SIZE 3257064 BODY[] {3257064}
> 2010-01-10 19:30:45.868000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:STREAM:OPEN Size: 3257064: Begin Message Download Stream
> 2010-01-10 19:30:58.446000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:STREAM:CLOSE: Normal Message End Download Stream
> 2010-01-10 19:30:58.447000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 8230 OK Success

(9) IDLE
> 2010-01-10 19:30:58.781000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 8231 IDLE
> 2010-01-10 19:30:58.953000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: + idling

(10) After 1 miutes, while idling at imapsamplemailbox, login for [Gmail]/Drafts occurs
> 2010-01-10 19:31:00.117000 UTC - 0[a2d140]: 8e31800:imap.googlemail.com:NA:SetupWithUrl: clearing IMAP_CONNECTION_IS_OPEN
> 2010-01-10 19:31:00.118000 UTC - 5932[5346a40]: ImapThreadMainLoop entering [this=8e31800]
> 2010-01-10 19:31:00.118000 UTC - 5932[5346a40]: 8e31800:imap.googlemail.com:NA:ProcessCurrentURL: entering
> 2010-01-10 19:31:00.118000 UTC - 5932[5346a40]: 8e31800:imap.googlemail.com:NA:ProcessCurrentURL:imap://[e-mail]:993/select%3E/%5BGmail%5D/Drafts:  = currentUrl
> 2010-01-10 19:31:00.880000 UTC - 5932[5346a40]: 8e31800:imap.googlemail.com:NA:CreateNewLineFromSocket: * OK Gimap ready for requests from 128.100.240.244 8if1458905yxj.52
> 2010-01-10 19:31:00.880000 UTC - 5932[5346a40]: 8e31800:imap.googlemail.com:NA:SendData: 1 capability
> 2010-01-10 19:31:00.947000 UTC - 5932[5346a40]: 8e31800:imap.googlemail.com:NA:CreateNewLineFromSocket: * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA XLIST CHILDREN XYZZY
> 2010-01-10 19:31:00.947000 UTC - 5932[5346a40]: 8e31800:imap.googlemail.com:NA:CreateNewLineFromSocket: 1 OK Thats all she wrote! 8if1458905yxj.52
> 2010-01-10 19:31:00.948000 UTC - 5932[5346a40]: 8e31800:imap.googlemail.com:NA:SendData: Logging suppressed for this command (it probably contained authentication information)
> 2010-01-10 19:31:01.489000 UTC - 5932[5346a40]: 8e31800:imap.googlemail.com:NA:CreateNewLineFromSocket: * CAPABILITY IMAP4rev1 UNSELECT LITERAL+ IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE
> 2010-01-10 19:31:01.489000 UTC - 5932[5346a40]: 8e31800:imap.googlemail.com:NA:CreateNewLineFromSocket: 2 OK [e-mail] authenticated (Success)
> 2010-01-10 19:31:01.489000 UTC - 5932[5346a40]: 8e31800:imap.googlemail.com:A:SendData: 3 COMPRESS DEFLATE
> 2010-01-10 19:31:01.687000 UTC - 5932[5346a40]: 8e31800:imap.googlemail.com:A:CreateNewLineFromSocket: 3 OK Success
> 2010-01-10 19:31:01.687000 UTC - 5932[5346a40]: 8e31800:imap.googlemail.com:A:SendData: 4 select "[Gmail]/Drafts"
> 2010-01-10 19:31:01.892000 UTC - 5932[5346a40]: 8e31800:imap.googlemail.com:A:CreateNewLineFromSocket: 4 OK [READ-WRITE] [Gmail]/Drafts selected. (Success)

(11) After 2 minutes from start of IDLE, enter idling again
> 2010-01-10 19:32:41.905000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: DONE
> 2010-01-10 19:32:42.073000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 8231 OK IDLE terminated (Success)
> 2010-01-10 19:32:42.073000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:ProcessCurrentURL: entering
> 2010-01-10 19:32:42.073000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:ProcessCurrentURL:imap://[e-mail]:993/folderstatus%3E/imapsamplemailbox:  = currentUrl
> 2010-01-10 19:32:42.073000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 8232 noop
> 2010-01-10 19:32:42.255000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 8232 OK Success
> 2010-01-10 19:32:42.260000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 8233 IDLE
> 2010-01-10 19:32:42.430000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: + idling

(12) After 5 minutes, unsol "* 3099 EXISTS". Gmail looks to send unsol response each 5 minutes.
> 2010-01-10 19:37:47.573000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * 3099 EXISTS
> 2010-01-10 19:37:47.723000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: DONE
> 2010-01-10 19:37:47.899000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 8233 OK IDLE terminated (Success)
> 2010-01-10 19:37:47.899000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:ProcessCurrentURL: entering
> 2010-01-10 19:37:47.900000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:ProcessCurrentURL:imap://[e-mail]:993/select%3E/imapsamplemailbox:  = currentUrl
> 2010-01-10 19:37:47.900000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 8234 check
> 2010-01-10 19:37:48.071000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 8234 OK Success
> 2010-01-10 19:37:48.072000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 8235 getquotaroot "imapsamplemailbox"
> 2010-01-10 19:37:48.252000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * QUOTAROOT "imapsamplemailbox" ""
> 2010-01-10 19:37:48.252000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * QUOTA "" (STORAGE 65702 7600203)
> 2010-01-10 19:37:48.252000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 8235 OK Success
> 2010-01-10 19:37:48.252000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 8236 UID fetch 3180:* (FLAGS)
> 2010-01-10 19:37:48.436000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * 3099 FETCH (UID 3179 FLAGS (NonJunk))
> 2010-01-10 19:37:48.436000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 8236 OK Success

(13) Enter idling again
> 2010-01-10 19:37:48.485000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 8237 IDLE
> 2010-01-10 19:37:48.663000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: + idling

(14) After 5 minutes, unsol "* 3099 EXISTS". Gmail looks to send unsol response each 5 minutes.
> 2010-01-10 19:42:53.816000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * 3099 EXISTS
> 2010-01-10 19:42:53.817000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: DONE
> 2010-01-10 19:42:53.986000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 8237 OK IDLE terminated (Success)
> 2010-01-10 19:42:53.986000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:ProcessCurrentURL: entering
> 2010-01-10 19:42:53.986000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:ProcessCurrentURL:imap://[e-mail]:993/select%3E/imapsamplemailbox:  = currentUrl
> 2010-01-10 19:42:53.986000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 8238 noop
> 2010-01-10 19:42:54.154000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 8238 OK Success
> 2010-01-10 19:42:54.155000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 8239 getquotaroot "imapsamplemailbox"
> 2010-01-10 19:42:54.396000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * QUOTAROOT "imapsamplemailbox" ""
> 2010-01-10 19:42:54.396000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * QUOTA "" (STORAGE 65702 7600204)
> 2010-01-10 19:42:54.397000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 8239 OK Success
> 2010-01-10 19:42:54.397000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 8240 UID fetch 3180:* (FLAGS)
> 2010-01-10 19:42:54.592000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: * 3099 FETCH (UID 3179 FLAGS (NonJunk))
> 2010-01-10 19:42:54.592000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: 8240 OK Success

(15) Enter idling again
> 2010-01-10 19:42:54.671000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 8241 IDLE
> 2010-01-10 19:42:54.846000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:CreateNewLineFromSocket: + idling

(16) Termination of Tb
> 2010-01-10 19:45:42.295000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: DONE
> 2010-01-10 19:45:42.296000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 8242 close
> 2010-01-10 19:45:42.296000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:SendData: 8243 logout
> 2010-01-10 19:45:42.326000 UTC - 5932[5346a40]: ImapThreadMainLoop leaving [this=8e31800]
> 2010-01-10 19:45:42.326000 UTC - 4472[5346540]: ImapThreadMainLoop leaving [this=800f400]
> 2010-01-10 19:45:42.326000 UTC - 5664[5345640]: 73e0800:imap.googlemail.com:S-imapsamplemailbox:TellThreadToDie: close socket connection
> 2010-01-10 19:45:42.326000 UTC - 5664[5345640]: ImapThreadMainLoop leaving [this=73e0800]
> 2010-01-10 19:45:42.328000 UTC - 6100[5347800]: ImapThreadMainLoop leaving [this=73e3c00]
> 2010-01-10 19:45:42.328000 UTC - 3592[53458c0]: ImapThreadMainLoop leaving [this=5cba000]

Comment 21

[WADA:World Anti-bad-Duping Agency]

For your case.

No connection error existed in your case, because any request/response to/from IMAP server is normally written in the IMAP log after last append command. So, if problem exists in copy process(number of appended mail is less than number of requested mail), it's never due to issue like connection loss while IMAP processing in your case.  

As "* 3099 EXISTS" is returned from server to last append(UID=3179) command and while idling, actual number of mails in imapsamplemailbox is 3099.
As seen in Attachment #586633 [details], UID=1 to UID=3179 is added by append command in your case(fortunately, Gmail IMAP supports UIDPLUS.) 
These means number of appended mails was 3179, and, before appending, no mail existed in imapsamplemailbox at server(it's correct because you created the IMAP folder just before test.) And difference between 3179(highest UID) and 3099(number of mails in the folder) is due to duplicated mails in your case.

Where does "5142" in your "It successfully copied 3099 of 5142 messages before it stopped" come from?
5142 mails actually existed in move source folder(local mail folder) before you requested move but append was requested by Tb for only 3172 mails when you got the IMAP log?

If Tb 9, log like "CopyNextStreamMessage: Copying 8 of 10000" is written by Tb in IMAP log. If possible, execute duplucation test with Tb 9, please, with stopping auto mail check of any IMAP mail folder to exclude unwanted log for [Gmail]/All Mail.
(After removal of log for [Gmail]/All Mail, your 800MB log file was reduced to 240MB. Until removal of log for [Gmail]/All Mail, log analysis was nearly impossible due to too large log file and excess log lines for [Gmail]/All Mail, even though Script was used to extract required log lines.)

Because "move mails" == "copy mails to target folder"(bulk copy) + "delete mails from source folder"(mass delete), delete step of "move many mails" is highly affected by problem like bug 538378.
Is there no problem in "manual delete of 3179 mails" or "manual delete of 5142 mails" at the local mail folder?

By the way, because Gmail IMAP is special IMAP as seen in "upload of dup mails" case, please be careful in testing and reporting Tb's bug at B.M.O.
If problem was "only 3099 mails in Gmail IMAP folder after move of 3179 mails in local folder", it's apparently INVALID in your case.

Comment 22

[WADA:World Anti-bad-Duping Agency]

(In reply to Magritte from comment #5)
> What's worse is at one point I accidentally moved the messages from the
> local folder to the IMAP folder while Thunderbird was in offline mode. This
> caused all the messages to be deleted from the local folder and appear in
> the IMAP folder.

This is already improved by recent Tb such as Tb 9.
For both offline-use=on folder and both offline-use=off folder, offline-store file is used for move/copy to IMAP folder while Work Offline mode. And moved mails are not deleted from local mail folder by the move operation while Work Offline mode.
When go back to Online mode, copied mails at thread pane of the IMAP folder disappears at thread pane once(when first uid fetch 1:* flags, temporary uid of copied mails doesn't exists, so disappears), then appended to the IMAP mbox, and by re-syc of mails, all copied mails are shown again.
After successfull upload of copied mails, mails in move source folder(local folder) is deleted.
So, "move from local to IMAP while Offline mode" is already safer than before.

> However, at least in some cases there does appear to be a potential for serious
> data loss due to this bug if a large move is attempted while in offline mode.

Move/delete while Work Offline mode still has many issues.
  Even in Tb 9, mismatch for user surely exists :
    Mails are moved to other IMAP folder, but they still remain in local mail
    folder, even though user *MOVED" mails.   
  If compact happens on local/move-source mail folder, delete from the local
  folder perhaps fails to find the moved mails, because offset of mails are
  changed by Compact.
  If multiple Thunderbird accesse same IMAP mbox, any Tb can go Offline mode
  and any Tb can do move of mails between IMAP folders during Offline mode.
  In such case, how consistency of multiple mail moves at multiple Tb intances
  can be kept, without any confusion by users?
  (Work Offline mode of Tb is not designed for shared access of IMAP Mbox)
  (by multiple Tb instances, but many users actually shares an IMAP Mbox )
  (by multiple IMAP clients including multiple Tb instances.             )
Some of above are perhaps already reported to other bug.
Please open separate bug for issues around "Work Offline mode", after search for already opened bugs for it well.
Please keep "one prolem per a bug".

Comment 23

[WADA:World Anti-bad-Duping Agency]

I believe this bug is for next;
(a) Copy step of move mails(append) looks to stop somehow at mid of copy of mails,
    even though IMAP level error/TCP level error never occurs.
    Number of appended mails < Number of mails move is requested.
(b) And, it seems that UI freezes for a while.
(c) Even after UI freeze disappears, nothing happen on move source folder.
    At Tb window where move mails is requested, selection of all mails is still
    kept.
(d) No mail is deleted from move source folder even though some of requested
    mails are already uploaded to server successfully.

I guess (b) is due to fetch of all appended mails from [Gmail]/All Mail which is executed concurrently.
You probably didn't disable auto mail check or you opened [Gmail]/All Mail with DLE enabled when you tested, with setting offline-use=on for Gmail IMAP folder(auto-sync tries to whole mail data of all mails when new mail is detected).
Because Gmail IMAP holds data of all mails in [Gmail]/All Mail, append of a mail to imapsamplemailbox generates a new mail in [Gmail]/All Mail. If new mail check is done or if the new mail is notified via IDLE, Tb knows about the new mail and Tb invokes auto-sync process for [Gmail]/All Mail sooner or later, concurrently with append process for "move mails".
And, if offline-use=on is set for imapsamplemailbox, auto-sync for imapsamplemailbox is invoked concurrently with with append process for "move mails", because IMAP connection for the imapsamplemailbox is not locked for "move mails" process only.
Because auto-sync is "download of whole mail data", it takes long, and because IMAP task is currently executed under UI process, short freeze of UI may occur in your test.

(c) is same as phenomenon after connection loss while appending mails for bulk move. Because no connection error happened in your case, issue of (a) is perhaps cause of this phenomenon.

(d) is probably a simple result of "current move mails" = "bulk copy" + "mass delete".  

If (a) actually occurred in your case, I guess contention between next processes is relevant to problem.
(1) Continuous "append" for mails requested to move.
(2) Header fetch for newlly appended mails.
(3) New mail check when unsol "* nnn EXIST" while idling.
    (Gmail IMAP looks to send it each 5 minutes.
(4) New mail check by enabled "automatic new mail check" for the IMAP folder.
(5) Auto-sync for the IMAP folder, because Tb knows about new mails by (1) to (49.
(6) Filter copy/move to the IMAP folder can occur if new mail check is enabled.
(7) MsgPurge by retention policy may occur on the IMAP folder at same time.
(8) Auto-compact may participate to the contention.

Even though "use of connection for the IMAP folder" is properly serialized by tasks like above, tasks for above may interfere each other.
As seen in my IMAP log, log of "CopyNextStreamMessage: Copying NNN of 10000" is written after successful append of NNN-th mail.
The log is written by next code.
> http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapMailFolder.cpp#6748
> 6731 nsImapMailFolder::CopyNextStreamMessage(bool copySucceeded, nsISupports *copyState)
> 6732 {
> 6733   //if copy has failed it could be either user interrupted it or for some other reason
> 6734   //don't do any subsequent copies or delete src messages if it is move
> 6735   if (!copySucceeded)
> 6736     return NS_OK;
> 6737   nsresult rv;
> 6738   nsCOMPtr<nsImapMailCopyState> mailCopyState = do_QueryInterface(copyState, &rv);
> 6739   if (NS_FAILED(rv))
> 6740   {
> 6741     PR_LOG(IMAP, PR_LOG_ALWAYS, ("QI copyState failed:%lx\n", rv));
> 6742     return rv; // this can fail...
> 6743   }
> 6744 
> 6745   if (!mailCopyState->m_streamCopy)
> 6746     return NS_OK;
> 6747 
> 6748   PR_LOG(IMAP, PR_LOG_ALWAYS, ("CopyNextStreamMessage: Copying %ld of %ld\n", mailCopyState->m_curIndex, mailCopyState->m_totalCount));

Because there is no log for next append after last append(for UID=3179) in your IMAP log, it looks for me that next copy(append) was not requested to IMAP code due to one of unexpected conditions in above code.
"Connection loss while appending" perhaps produces one of unexpected conditions in above code.

As written in comment of above code, Tb stops further "move mails" process if such condition occurs.
So, cause of such condition is clear in connection loss case.
What did produce such condition in your case?

Comment 24

[WADA:World Anti-bad-Duping Agency]

FYI.
I've opened bug 716461 for phenomenon of comment #18 and comment #19 when LAN cable is pulled off while uploading mails to IMAP server.

Comment 25

[WADA:World Anti-bad-Duping Agency]

Bug 717016 may be relivant to your problem.
> Bug 717016 pending URI queue management is problematic

Another concern, Auto-Compact, because move/copy from local mail folder.
Mail in local mail folder is pointed by url style path like next. 
                    +-- mail.server.serverN.userName
                    |   +-- mail.server.serverN.hostname
                    |   |     +-- folder name
                    |   |     |     +-- offset of mail data in mail folder file
                    V   V     V     V
  mailbox-message://x@x.x.x/Inbox#23579
(a) If this offset value is obtained upon "Select All" and is passed to "Move/Copy" task at once, and (b) if Auto-Compact occurs while move/copy is running and offset of mails is changed, and (c) if offset change is not notified to move/copy task or move/copy task doesn't care for offset change notification, (d) move/copy may fail due to offset change(mail is not found at the requested offset).
And, (e) if a mail is shifted by Auto-Compact to known offset of other mail, and (f) if length of the shifted mail is same as the known other mail(already shifted to different offset by Compact), (g) wrong mail may be uploaded(or wrong mail is deleted if delete step of Move).

When "delete of old draft by save as draft", (d) in above is "(h) old draft is not deleted" and (g) in above is "(i) wrong draft is deleted", and both (h) & (i) actually occurs in Tb. So, I think (d) & (g) can occur.
If (d) occurs, move/copy of subsequent mails may continue, but move/copy of subsequent mails may stop.

Comment 26

[Andreas Topp]

i can wath this problem for years now! the problem, that copy a lot of mails including folders and subfolders stops without any error, is present for years now! i had never success with moving larger amount of mails or folders from one to another (not-local) mailbox. that is very painful! 

also, for example, empty trash is not showing in status-bar while it is still working somehow! when i empty the trash which contains many folders (i deleted them, because the copy-process was not completed correctly), i can see that every second or something like that, there is deleted one folder after another on my imap-server, long time ago after the deleting-process was done in thunderbird.

this makes thunderbird in this situation totally useless! i was searching for addons (import-export, copy folder) but they dont help me either. import-export does not give an option to export to filesystem while keeping folder-structure, copy folder has the same problem than thunderbird by itselfe. there is no way to get the mails from the local pop3-mailbox to the new imap-mailbox... i could copy them directly with scp and put them at the right place on our mailserver, but because i cannot export to filesystem with keeping folder-structure this is impossible.

...it would be a great help, if this problem could be really fixed in the future! 
bug 601900 was about the same problem and the bug was closed as fixed, but it isn't till today!

please, dear thunderbird team, it would be a very great help to see this fixed! 
this is more important then releasing one version and new feature after another!

if i can help you somehow with tests and logfiles, please let me know!

Comment 27

[Andreas Topp]

some additional: right now i copy about 100 folders from a customer "by hand" - means i select 3 folders and move them by drag&drop to the imap-mailbox. 
and even with that small amount (the folders does contain only 10-20 mails for example) the process fails sometimes and only the first folder contains mails...

Comment 28

[DT]

This problem is still present now and in my case could be described as follows:
Using Thunderbird connected by IMAP to two accounts.
One Account is a Microsoft Exchange Account.
One Account is a Gmail Account.
Task: Copy all the contents of a message folder containing about 3000 emails from the Gmail Account to the Exchange Account.
Method tried:  Highlight all the emails. Right click and select Copy To and select the Folder in the Exchange Account.
Result: Only a few emails are copied.
If repeat then destination shows duplicates of same emails.

Is there any solution?

Comment 29

[WADA:World Anti-bad-Duping Agency]

(In reply to DT from comment #28)
> Using Thunderbird connected by IMAP to two accounts.
> One Account is a Microsoft Exchange Account.
> One Account is a Gmail Account.
> Task: Copy all the contents of a message folder containing about 3000 emails
> from the Gmail Account to the Exchange Account.
> Method tried:  Highlight all the emails. Right click and select Copy To and
> select the Folder in the Exchange Account.
> Result: Only a few emails are copied.

Did you get IMAP log and check IMAP level flow?
All mails were normally uploaded to MS Exchange server(append mail data), but only two mails are shown as "number of mails" or only two mails was shown at Thread pane, weren't it?  

> If repeat then destination shows duplicates of same emails.

Did you actually check actual mail number in Mbox of MS Exchange server after first "copy of all mails", before second "copy of all mails"?

Did "network error" or "NO response to append comman" etc. occur during first ""copy of all mails"?
Is "copy source Mbox of Gmail" and "copy target box of MS Exchange" Offline-UseOn folder?
If Offline-Use=On folder and some error happemed during first "copy of all mails", it may be following.
1. first copy is executed with utilizing Offline-Store file.
2. After copy of two mail, some eroor happened.
3. Before execution "append for second copy of all mails", uploading for failed mails whlee "first copy of all mails" is retried, and is normally uploaded.
4. Second "copy of all mails" is normally executed.

Comment 30

[Andreas Topp]

some additional information: after doing the hand-copy-job a hole day thunderbird was not able to copy even a single mail! we found out that the cause of this was a full hdd! 

you should implement some error-recognition for the move-command and catch this with try{} catch{} ...

this should be the first step for this problem, or not?

of course this is not the only reason, because in all other cases, the hdd had have enough space...

Comment 31

[Michael Goodwind]

Same problem over here and considering how important this functionality could be it is absolutely mind boggling and frustrating to say the least that it hasn't been fixed in over 4 years.  

I am experiencing the exact symptoms as the above mentioned posts and problem persists in tb 24.5.0

Comment 32

[Alex]

It seems to be a problem with mails that have an attachment of more than 5MB.

Comment 33

[Andreas Topp]

Hey all, don't forget to vote for this bug if you are interested in a solution! I think, this is very important in order to increase the probability, that someone will work on this!

Comment 34

[eundas]

I'm having the same problems described above when trying to move emails between a set of local folders and an IMAP email account. However, I see this happening also for small numbers of emails. The only case in which it doesn't fail is if I move all the messages one by one, which is obviously not practical. If I drag (or copy, or move, etc.) any known number of emails, in many cases only a subset of the selected emails actually reaches the destination. No warnings or errors are generated so unless one is carefully checking the result of the operation, the risk of missing messages is very high.

Comment 35

[Joel Nelson]

1) My platform is Linux.
2) I'm copying from IMAP to local folder.
3) Small numbers (25 or so) seem to copy fine.

Comment 36

[Joel Nelson]

Also, My version of Thunderbird is 31.2.0

Comment 37

[AL]

Solution:

I had similar problem and it seems to be solved now by the following:
- In Tools > Options > Advanced > Disk space, I increased cash to max 1024 MB.
- In Tools > Options > Advanced > General, I opened Config Editor, I searched for 'timeout' and I increased usually by x10 the time settings for about 20 positions which to my naive understanding seemed relevant.

Comment 38

[AL]

Solution (additional info):
After making changes, restart Thunderbird.

Comment 39

[Andreas Topp]

AL, thanks for your answer.

Anyway, this is not an solution, it is an workaround! the System should show a error-message if there is a problem while transfer mails from one to another account!

the bug is open for over 5 years, has over 6 votes right now and still nobody is working on it - would be nice if this would change in nearby future ;-)

Comment 40

[maherik]

Confirming that this extremely annoying bug still exists in Thunderbird 45.4.0 running on Win7 x64.  Discovered bug and found this thread after attempting to migrate multiple Yahoo accounts to Outlook and Gmail after recently hearing of the massive hacks on Yahoo in 2014 and trying to shut down all of my Yahoo accounts.  However, this bug is not specific to Yahoo.  Moving large quantities of Gmail to Outlook or Outlook to Gmail produces the same result:  Thunderbird stops the move midway with no warning or error message and then goes into a kind of "limp mode" where any successive attempts to move anything cease to work - and don't even report any progress - until I close and restart Thunderbird and try again.  Upon doing so, the move operation fails again and again.  There are spurious copies of emails in the source folder each of which should have been deleted during its move to the destination folder, but Thunderbird won't delete anything until the entire move operation completes successfully, which never happens.

Comment 41

[Hans]

Same problem over here with Thunderbird 45.5.0 and Mac OSX 10.11.6. 

Need to move 2000 messages from one IMAP account to another. Not possible to move more than 100-300 at once, then the process fails. Very time intensive and frustrating!

Comment 42

[Wayne Mery (:wsmwk)]

others related to imap -> local folders: bug 842267, bug 1272681, bug 763390, bug 1290669, Bug 775008

Comment 43

[Wayne Mery (:wsmwk)]

and bug 1115328,

Comment 44

[José Ángel Morente]

Can't believe it!  This bug is open more than 8 years and still no solution.
I stopped using Thunderbird because of this failure, but now I've had to use TB again at work and the problem is still here!

Please, someone should fix this!  It would be very nice if at least you could continue the operation where it previously failed, but it's impossible because it does not delete e-mails that already have been copied to the IMAP folder. That's insane!  The atomic operation here should be moving ONE e-mail, so after it is copied to the IMAP folder it should be removed from the source folder so in case of failure you can just move the rest of e-mails.

Comment 45

[Markus Mobius]

Your mileage might vary - but I found that when I change the focus on another folder and then back it continues again.

Comment 46

[Todd]

I now have three customers with this issue.  WHAT A PAIN IN THE A**.  Please fix this as soon as possible!  It is an emergency for these customers.

Comment 47

[Todd]

Work around: on one customer, I renamed all her sub folders in Local folder to remove the spaces and replace them with underscores.  The transfer would mysteriously stop if it came across a folder with spaces in its name

Comment 48

[David Martin]

I have this problem copying a block of messages from tuffmail to outlook.com. The folders don't have spaces in the names.

Comment 49

[Sans]

I am TRYING to copy a 27,000 email folder to a gmail account using imap and the process fails at random point with no error message whatsoever.... no possibility to resume, etc.....
mind boggling to see this after 9 YEARS.

Comment 50

[John Poole]

I've suddenly started having this problem, too.  100-300 emails.  This has worked for over a decade.  I'm wondering if the problem lies on the IMAP end.

Comment 51

[José Ángel Morente]

(In reply to John Poole from comment #50)
> I've suddenly started having this problem, too.  100-300 emails.  This has
> worked for over a decade.  I'm wondering if the problem lies on the IMAP end.

Even if the problem is on the IMAP side, Thunderbird should handle the problem the right way, but it doesn't. It just stops the moving operation with no warning, and no chances to resume the job (you need to manually compare thousands of folders to get what is missing). Just unacceptable. This issue was reported almost 10 years ago and it still remains unsolved. I and my company stopped using Thunderbird because of this issue.

Comment 52

[Todd]

This is driving me INSANE. What human ritual do I need to preform to get you guys to fix this?

In the mean time, anyone know how to off line restore imap directories from backup?

Comment 53

[Comfine GmbH]

Todd, what do you mean with "offline" and what kind of backups do you have? .... anyway, i think, thats the wrong place to discuss that offtopic.

but hey, Thunderbird-Team, you got new programmers, right? So this is really a major-bug and you really should focus on it! :-)

Comment 54

[Wayne Mery (:wsmwk)]

(In reply to Todd from comment #52)

In the mean time, anyone know how to off line restore imap directories from backup?

Please let's not cover support questions in bugzilla - please post in https://support.mozilla.org/en-US/questions/thunderbird

Comment 55

[Todd]

(In reply to Wayne Mery (:wsmwk) from comment #54)

Todd, what do you mean with "offline" and what kind of backups do you have? .

Answered to Wayne's eMail address

Comment 56

[antonioaltamura7]

wait are you telling me with bug hasn't been solved in 10 years?

Comment 57

[Wayne Mery (:wsmwk)]

And More fun

Comment 58

[gene smith]

Well, I "copied" a folder with about 1600 emails to a gmail folder and only copied about 900 messages one time. Then I deleted the messages from the gmail folder and tried again. This time it only copied about 700.

Since I don't run this laptop with offline storage, it appears that the messages are successively fetched from the source imap folder. Then at some point there is a problem and in the status bar I see something about trying to reconnect. At that point I think the partial folder content, which is probably concatenated into an mbox-like temporary file, is imap appended to the gmail folder, which happens quickly. I need to look closer at what is happening using imap:5 log and/or wireshark, but in any case, there is definitely a problem here and I understand all the complaints.

Edit: Doing this while recording IMAP:5 log, I see that the each email is fetched from the source folder in ascending UID order and immediately appended to the destination folder. So the order is fetch, append, fetch, append ...
Looking at the log, I see that somehow the connection to the source server is lost. Can't tell from the log why or who is initiating the disconnect, tb or the server. This time over 1000 emails were fetch and appended before the problem occurred.
I then transferred a small folder having offline store to gmail. With source offline store, nothing is fetched from the source server. Each message is just read from the mbox file and individually appended. So there is less to go wrong when you have offline store.
Anyhow, given this info, I need to peruse WADA's detailed posts above and see what they say.

Comment 59

[gene smith]

Someone reported above that increasing timeouts helped. However, he didn't say which one. Also, reading some of WADA's comments mentioned bad effects of various background activities like periodic checks for new mail and autosync. So on all accounts I disabled checks for new mail and immediate notification for new mail and in preferences (config ed.) I disabled autosync. With these changes and tb restart to ensure they take effect, the 1600 emails copied OK. Not sure if this is just luck so I need to try the copy a few more times...

Comment 60

[José Ángel Morente]

(In reply to gene smith from comment #59)

Someone reported above that increasing timeouts helped. However, he didn't say which one. Also, reading some of WADA's comments mentioned bad effects of various background activities like periodic checks for new mail and autosync. So on all accounts I disabled checks for new mail and immediate notification for new mail and in preferences (config ed.) I disabled autosync. With these changes and tb restart to ensure they take effect, the 1600 emails copied OK. Not sure if this is just luck so I need to try the copy a few more times...

IMHO it has been just luck. Anyway, the main issue here is not the process being interrupted; that could be acceptable under certain circumstances. The root of the problem is that TB does NOT warn correctly about the connection drop and, furthermore, it hasn't a way to resume the operation nor a proper rollback to get things just like they were before the failure.

Furthermore, 1600 is almost nothing if compared to real life. A typical mailbox (in the business environment) nowdays can have 20,000 emails per folder. and a hierarchy 300+ folders. ThunderBird is just a no-go for most enterprise users because its inability to properly deal with big amount of data.

Comment 61

[gene smith]

Well, just did the 4th time with no errors so doesn't seem like luck, but who knows. Before it failed (didn't copy all 1600) 3 times in a row. Anyhow, this is complicated stuff so I can only do one step at a time...
Curious how you copy 300+ folders? All I see is you select one folder and copy or move the message in the folder to another folder on another server. Is there a way to copy or move 300 folders in one command that I don't know about in tb? Maybe an extension?
I really don't know how tb stacks up as in business environments since I just volunteered to help out with imap issues. All of it was designed by paid mozilla people who are now long gone and apparently even 10 years ago when some were still around there wasn't much concern about this issue.

Comment 62

[José Ángel Morente]

Copying 300+ is a pain in the neck because it must be done one by one, so if you have to do 300 operations and it fails 200 times, it's an endless task!

Comment 63

[Comfine GmbH]

gene, if you copy a folder normally all subfolders and included mails in this folders will be copied aswell, so the "command" to copy more then one folder is "built-in".

Comment 64

[gene smith]

(In reply to Comfine GmbH from comment #63)

gene, if you copy a folder normally all subfolders and included mails in this folders will be copied as well, so the "command" to copy more then one folder is "built-in".

I see. There doesn't seem to be a menu command or right-click command to copy/move a folder but you can select a folder or multiple folders with ctrl click and drag it/them to do the move. You can also drag with ctrl pressed to just copy the folder(s).

I tried my test with 7000+ messages moved twice with no problem (all messages transferred from server A to server B and back to server A). I will repeat the test an go out of wifi range and see what happens.

Comment 65

[José Ángel Morente]

(In reply to gene smith from comment #64)

(In reply to Comfine GmbH from comment #63)

gene, if you copy a folder normally all subfolders and included mails in this folders will be copied as well, so the "command" to copy more then one folder is "built-in".

I see. There doesn't seem to be a menu command or right-click command to copy/move a folder but you can select a folder or multiple folders with ctrl click and drag it/them to do the move. You can also drag with ctrl pressed to just copy the folder(s).

I tried my test with 7000+ messages moved twice with no problem (all messages transferred from server A to server B and back to server A). I will repeat the test an go out of wifi range and see what happens.

Just reproduce the failure by disabling the network some seconds. The problem is not the interruption itself, but the absolute lack of information when that occurs and a usable mechanism to resume the operation.

Comment 66

[gene smith]

With weaker wifi, my copies don't work quite as well. This summarizes what I see in the log when the copy from source server to destination server finishes prematurely:

Many fetch/appends occur and succeed, ie, messages copied from source to destination.
Append fails at at destination. Connection was dropped per log.
Tb makes new connection to retry the queued "append from file" url.
Append retry succeeds
Log message occurs: [30784:Main Thread]: I/IMAP CopyNextStreamMessage: Copying 458 of 7651
Fetch next message from source fails. Connection dropped.
Tb makes new connection to source to retry the queued "fetch" url.
Fetch retry succeeds.
Connection to destination reports that it has dropped. This is before the append has started.
New connection to destination made. "Append from file" url was never queued, per the log.
Tb fetches headers for all the emails in destination mailbox and seems to conclude, erroneously, the "copy" operation is done.
Only 457 messages copied and no error notification occurs.

I see this with laptop setting outside under trees. Don't have debugger or network sniffer. Also, this is still with autosync and checks for new mail disabled.

Comment 67

[Wayne Mery (:wsmwk)]

This and bug 720158 are commonly reported, so -> critical

Comment 68

[gene smith]

(In reply to Wayne Mery (:wsmwk) from comment #67)

This and bug 720158 are commonly reported, so -> critical

bug 720158 also reports a hang. I never see a hang or tb lockup, just tb gets an unrecoverable network error and stops the copy and moves on like everything is OK.

I tested this for several days on two systems and usually the copy (7000+ messages) worked OK but sometimes not. What I think happens is that the destination or source IMAP server does not expect so much data for so long a time from tb and drops the connection(s) and refuses new connections. This prevent any retries from being effective. This may depend on how busy the server is since I noticed that the time of day made a difference (afternoon bad, evening better).

A major complaint above is that it is difficult to recover from the error without completely starting again. One way I found is to sort the source and destination folders by "Order Received". Tb copies in ascending UID order so if a partial copy occurs, you can look in the destination folder and find the highest numbered "Order Received" value (really the message UID) and note the subject and date. Then go to the source folder and copy one past that same message to the end (highest Order Received/UID). This will avoid duplication and having to completely start again.

Note also that when tb does a "move" to another server, nothing is marked \deleted unless the move 100% succeeds and completes. Then all the message in the source are marked \deleted. Of course, this assumes you are using move mode and not copy.

Disabling autosync and checks for new mail while doing this also seems to help, see comment 59.

Another complaint is that there is no indication when copy/move is done or if it has stopped due to failure. A pretty good indication of completion is that the status bar goes blank and shows no activity. Of course this still requires the user to check the destinations for complete copies or incomplete copies.

Also mention is problems for business or enterprise users. I tend to think that the sysadmin might use a dedicated 3rd party tool (e.g., https://imapsync.lamiral.info/) that transfers and sets up all the employee's folders at the new server for them rather than having them each copy or move 20,000 messages in 300 folder, for example, manually in tb.

In conclusion, several people have commented on this bug and state that is a reason to not use TB. I don't think for most users this is a showstopper (critical) since it is a pretty rare event to need to move messages to a new server and most user can do this if it's necessary and work-around the problems. Implementing a "restart" mechanism is probably very difficult. However, it seems like popping up an error message that the copy or move did not succeed should be fairly easy and don't know why that doesn't currently occur.

Comment 69

[Markus Mobius]

I frequently move data from one email account to another rather than copy.

It would have helped a lot if TB would remove copied messages along the way and not just at the end once the copying completed successfully - that way you would know how much was moved even when it fails.

I use it with Gmail where moving something twice doesn't result in duplicates. So I worked around this bug by moving emails in increments of 1000 emails or so at a time. When it completes these 1000 emails are removed from the source and I do the next batch. If it fails I just repeat it.

Still a pain but it worked to move my daughters' school accounts after they changed schools.

Comment 70

[José Ángel Morente]

(In reply to gene smith from comment #68)

(In reply to Wayne Mery (:wsmwk) from comment #67)

I tested this for several days on two systems and usually the copy (7000+ messages) worked OK but sometimes not. What I think happens is that the destination or source IMAP server does not expect so much data for so long a time from tb and drops the connection(s) and refuses new connections. This prevent any retries from being effective. This may depend on how busy the server is since I noticed that the time of day made a difference (afternoon bad, evening better).

A major complaint above is that it is difficult to recover from the error without completely starting again. One way I found is to sort the source

Difficult? Nah. It is impossible.
In the real world some users have thousands of folders in complex hierarchies of dozens of depth levels.

Disabling autosync and checks for new mail while doing this also seems to help, see comment 59.

Why should users disable autosync for daily and trivial operations?

Another complaint is that there is no indication when copy/move is done or if it has stopped due to failure. A pretty good indication of completion is that the status bar goes blank and shows no activity. Of course this still requires the user to check the destinations for complete copies or incomplete copies.

Again: when you have 1,000 folders in a chaotic hierarchy that's not an option.

If only TB deleted the emails that have been sucessfully moved... it would be easy to retry and finished the remaining emails. But since the deletion is done only once the whole operation is finished, at the end of the day you get lots of duplicate stuff across the source and target folders.

Also mention is problems for business or enterprise users. I tend to think that the sysadmin might use a dedicated 3rd party tool (e.g., https://imapsync.lamiral.info/) that transfers and sets up all the employee's folders at the new server for them rather than having them each copy or move 20,000 messages in 300 folder, for example, manually in tb.

Yeah, you got it. Many of us have started using a third party tool. It's called Microsoft Outlook.
Moving a big folder to another place is something a regular user should be able to do; definitely not a sysadmin task.

In conclusion, several people have commented on this bug and state that is a reason to not use TB. I don't think for most users this is a

Ok, if you don't think it's a showstopper we will continue using Outlook then.

Believe me... thousands of companies might have stopped using Thunderbird because of these kind of issues. Thunderbird is not a reliable tool to be even considered in a professional environment. Just from my side, I had to remove Thunderbird at two different companies I've worked for; almost 2,000 users in total. Think about it.

Comment 71

[gene smith]

(In reply to Markus Mobius from comment #69)

I frequently move data from one email account to another rather than copy.

It would have helped a lot if TB would remove copied messages along the way and not just at the end once the copying completed successfully - that way you would know how much was moved even when it fails.

At one time, from what I have seen WADA and others post, drag/drop between servers always did a copy. Now it seems to default to move unless you do a ctrl-drag/drop. I suspect they are all set at the end to \deleted since there would have to be an imap transaction sent for each message to mark it deleted instead just one message at the end that sets all the messages to \deleted, so you have fewer "round trips".

Probably this was not designed with moving many and huge folders in mind.

I use it with Gmail where moving something twice doesn't result in duplicates. So I worked around this bug by moving emails in increments of 1000 emails or so at a time. When it completes these 1000 emails are removed from the source and I do the next batch. If it fails I just repeat it.

Still a pain but it worked to move my daughters' school accounts after they changed schools.

Sounds like a reasonable workaround.

Comment 72

[José Ángel Morente]

(In reply to gene smith from comment #71)

(In reply to Markus Mobius from comment #69)

Still a pain but it worked to move my daughters' school accounts after they changed schools.

Sounds like a reasonable workaround.

Yup. It's reasonable for a girl's school account. But not for a call center or a sales manager.

Comment 73

[gene smith]

(In reply to José Ángel Morente from comment #70)

Disabling autosync and checks for new mail while doing this also seems to help, see comment 59.

Why should users disable autosync for daily and trivial operations?

A single user moving thousands of messages and folder to another server is a daily operation?

If only TB deleted the emails that have been sucessfully moved... it would be easy to retry and finished the remaining emails. But since the deletion is done only once the whole operation is finished, at the end of the day you get lots of duplicate stuff across the source and target folders.

Maybe that could be done (mark as deleted when the append of messages completes). TBD.

Also mention is problems for business or enterprise users. I tend to think that the sysadmin might use a dedicated 3rd party tool (e.g., https://imapsync.lamiral.info/) that transfers and sets up all the employee's folders at the new server for them rather than having them each copy or move 20,000 messages in 300 folder, for example, manually in tb.

Yeah, you got it. Many of us have started using a third party tool. It's called Microsoft Outlook.
Moving a big folder to another place is something a regular user should be able to do; definitely not a sysadmin task.

In conclusion, several people have commented on this bug and state that is a reason to not use TB. I don't think for most users this is a

Ok, if you don't think it's a showstopper we will continue using Outlook then.

Believe me... thousands of companies might have stopped using Thunderbird because of these kind of issues. Thunderbird is not a reliable tool to be even considered in a professional environment. Just from my side, I had to remove Thunderbird at two different companies I've worked for; almost 2,000 users in total. Think about it.

Where I worked (retired now) Outlook was standardized on not because Thunderbird (actually Netscape back then) was so bad but because everybody else was moving to Microsoft and nobody got fired for using Microsoft. Only the company's Microsoft Exchange server was allowed for use so there was no reason to need to copy stuff to random imap servers. So not sure what you say is the main reason few companies use TB (at least here, maybe more in Europe)

So without major redesigned, things that can be fixed are maybe:
-Record an error if copy/move fails so something pops up and is recorded in activity manager.
-Mark successfully appended messages as deleted in the source folder when operation is move.

Comment 74

[Comfine GmbH]

comeon guys, what we dont need is discussion about if TB is used by a schoolgirl or a manager or a sysadmin, that want to move messages from A to B, it's pain in the ass if a operation just quit without errors before completion, regardless if you copy 20 mails or 20.000, for sure, but we should focus on the BUG here!
now as this bug was classified as critical, i hope, there is a big chance that this long story might end within 2019!
and i am sure that there is no need of a redesign. everything that is needed is a intelligent error-handling when working copy/move/delete mails via IMAP and the possibility to press a "try again" button to complete the operation, that has been started but not 100% finished. i am sure, that everyone from the TB-dev-team knows exactly what to do, if they will just focus on this bug!

Comment 75

[gene smith]

Just a correction to what I wrote in comment 71:

At one time, from what I have seen WADA and others post, drag/drop between servers always did a copy. Now it seems to default to move unless you do a ctrl-drag/drop. I suspect they are all set at the end to \deleted since there would have to be an imap transaction sent for each message to mark it deleted instead just one message at the end that sets all the messages to \deleted, so you have fewer "round trips".

I checked this again and drag/drop of an imap folder to a different imap account does a copy of the source folder and not a move. So you should never see messages inside the source folder or the source folder itself be removed. But if you select messages inside a folder and drag them (or use the right click menu to move them) to a different account, they are removed from the source folder (marked as imap \deleted) and the source folder is not removed.

Therefore, having messages "disappear" as an indication of the progress of a move is only possible (as currently designed) when multiple messages are selected in a single folder are moved.

Comment 76

[gene smith]

Maybe off topic but other problems observed:

1. Drag of a folder to another folder in the same account works: The source folder becomes a sub-folder of the destination folder. This is done simply using the imap "rename" command. However, ctrl-drag/drop seems to be a noop -- nothing happens. So it seems it is not possible to copy a folder using drag/drop within an account. (Of course, right-click copy on individual messages works within an account.)

2. Sometimes a dragging a folder between accounts is a noop -- nothing happens. Seems to be random and can only be fixed with a tb restart. Then the drag/drop works again. I've seen this with my trunk build on desktop system and with release 60.7.1 running on laptop.

Comment 77

[gene smith]

bug 601900 mentioned in comment 26 above seems to be the same problem as this bug. However, it has been closed as fixed and the patch addresses problems with a virus scanner grabbing the temp file containing the message to be appended at the destination server. Also, it only seemed to affect Windows systems. That may be why I rarely see this when testing on linux. Of course it doesn't address the secondary issues that several here have loudly mentioned.

Comment 78

[jdien07]

Referring to the problem described in the original post, it's not just Windows. I had the same problem on OS X (with Thunderbird 60). It's keeping me from switching to Thunderbird so it's definitely a critical bug. I'm using an iCloud account with IMAP. If I log into it using Thunderbird, I get the same problem when I try to migrate my large local mail folders to Thunderbird from Mail (I've been using e-mail for a long time). Small folders not a problem. I don't have a virus scanner. I can't really join the Thunderbird team until this is resolved. Meanwhile I'm looking at alternatives, which is too bad as I really do prefer Thunderbird but I can't afford to give up all my e-mail archives. I'm surprised that a bug of this magnitude hasn't been addressed in ten years. It's not confidence building.

Comment 79

[gene smith]

(In reply to jdien07 from comment #78)
I looked at this about 2 months ago and concluded that this is not a trivial thing to fix but it does need serious work. See my comments starting at comment 58.

Can you tell me your approximate folder structure and how many messages are in the various folders? Also, what procedure you used to set up thunderbird and copy messages from account A to account B? Are you wanting to move/copy messages from iCloud imap account to another account in tb?

You also mention local folder from Mail to Thunderbird. Not sure what that means or if it involves IMAP.

Comment 80

[jdien07]

Thanks so much for looking into this! So my starting point is I'm using Mail under OS X 10.14.6 and the iCloud mail service (IMAP). I'd like to immigrate to Linux so I'm considering Thunderbird as my new e-mail client. For this initial trial effort, I am trying to migrate to Thunderbird while still on OS X to minimize complications. I last looked into this in June with Thunderbird 60. It was quite a nightmare with it quitting partway through. I'd try doing it in batches of 1000 messages but somewhere along the way it'd mess up, often by duplicating all the messages I had just tried to copy so I had to start all over again.

In response to your query, I tried again, but this time with the new 68.0 release to provide you with up-to-date information. I'm pleased to say so far things are going better so the new update may have fixed things (mostly)! Just to be timely, here are my initial observations. I'm using a desktop connected via ethernet to a gigabit switch which in turn is connected to a 50mps Fios internet connection so wi-fi is not an issue, unlike some of the prior posts on this thread. I have some very large folders so I keep only the most important ones on iCloud. The rest are local. A typical local folder has 14,367 messages. It is the contents of a list serve that I follow and it is helpful for me (as a scientist) to have my own copy rather than to have to rely on the limited tools available on the web archive. So I have a copy of Thunderbird set up and it is working fine for the messages on iCloud. Since Thunderbird's import tools for Mail no longer work (they were not updated to keep up with changes in OS X), the only way for me to accomplish this migration is to first make a copy of this directory on iCloud and then from there to make a copy on Thunderbird's local installation. The first step of making a copy on iCloud using Mail went smoothly. For today's test, I am using Thunderbird to select all 14,367 messages in the GUI and then right-clicking and choosing "Copy To...Local Folder" to copy them into an empty local Thunderbird folder. Thunderbird says Downloading message ... of 14418 and went through the entire set without a hitch. It does seem to be having some trouble keeping synched with changes on the server when I then deleted the files and did some other things via Mail. More once I have a clearer sense of what is going on.

Comment 81

[gene smith]

jdien07, I don't know of any changes between 60 and 68 that would directly affect copying many files in one imap operation. The only network change I know of added a tcp keepalive feature: Bug 1535969. However, this wouldn't seem to affect the problem reports for this bug since large copies will cause continuous network activity with no intentional idle connections.

Anyhow, glad it seems to be working for you, at least so far. So let me know how it goes.

Comment 82

[jdien07]

The message count at the bottom of the Thunderbird window often undergoes prolonged pauses, presumably while it's doing some kind of cleanup or indexing. My guess is that prior to the keepalive addition it was timing out during these pauses, that in turn activated the bug you're examining, having to do with handling of errors. So it appears I'm good now. I appreciate your looking into this error handling bug. I expect I won't always be checking my e-mail under ideal conditions. Let me know if I can be of further assistance.

Comment 83

[gene smith]

Interesting theory on why keepalive might actually help.I'll do some experiments and see what is happening. But even if it helps, others have pointed out that there is still no notification to the user that the copy (or move) failed or a way to restart a failed copy without completely starting over and maybe duplicating already copied messages at the destination.

Comment 84

[gene smith]

jdien07, I do have one question. Can you tell me how many messages get copied before you see the "prolonged pauses"? Right now I have a test folder with about 7K messages and have done copies using it. Is that enough to see the pauses? (The messages are fairly short and from an old mailing list archive I found.)

Comment 85

[jdien07]

My impression was that it was about a thousand. Perhaps it was doing the compression thing? But I tried it again with another folder and this time instead of counting down smoothly it is giving the number of messages in increments of about one hundred so harder to judge. I make a screen recording I could send you. Also I could send you the listserve messages themselves as a test case. They're fully public.

Comment 86

[jdien07]

As a further possible test case, I'm currently trying to move the contents of an online folder with 40,0000 messages to a local folder. Thunderbird has already made a local copy of the online folder as far as I can tell, so this should be straightforward for TB; however, if I try to select and move more than about 1300 messages at a time (using the right-click method), it does nothing. So once again no error message. But at least there's no longer data corruption with this version of TB.

Comment 87

[gene smith]

It would seem easy for tb to copy 40,000 messages from a folder with offline storage to local folders since no network should be involved. What do you mean by "does nothing"?

Also I could send you the listserve messages themselves as a test case. They're fully public.

If you can zip them all into 1 file and attach to an email and send them to me, that would be good. However, I don't want to receive 40,000 emails :).

Comment 88

[jdien07]

I right click and then nothing further happens. The messages are still selected. There are no further messages in the status line. I leave and come back in an hour, still nothing different. With less than about 1350 messages, the status line immediately announces it is moving the files, provides a running count, and at the end of it they have been moved. I don't know how to confirm that TB really has made a copy offline of the iCloud messages, but looking inside the ImapMail directory of my Thunderbird Profiles directory, I do see a 1.34GB file with the name of the folder so I assume that is the offline copy. I'd be happy to zip the messages to you. I assume you mean you want me to zip all 40,000 and e-mail that one .zip file to you but you don't want me to forward all of them individually to you? :) I could zip the file that is in the ImapMail directory. Would 1.34GB bomb your e-mail account though? How about I send you a Box link?

Comment 89

[gene smith]

Edit: oops, I saw 1.34 and was thinking 1.34 MB. So that probably wouldn't work via email. Also, missed the "Box" link suggestion. Not sure what that is but probably better than trying to email a gig file. Thanks.
Sure go ahead and send it zipped. I always need more emails to bring tb to its knees. :) Send it to [e-mail] which is almost unused, storage wise. If that bounces we can try my default email I used for bugzilla. I've sent myself bigger emails. (Or if you have a file sharing site like dropbox or google drive, you could put it there and send me the link. I think firefox has something like that too, not sure.)

Comment 90

[gene smith]

Testing with jdien07's mbox file he sent me with about 17k messages, when doing a mass copy from Local Folders to a gmail imap folder, something seemingly random occurs. Each message is "appended" to the gmail mailbox and with the initial append request sent by tb there is a length, e.g., {1000} indicating 1000 byte are to be send by tb. Tb starts sending and sends 1000 bytes correctly but then 2 more bytes are sent. This causes gmail to respond with an error and the copy stops.
This seems to occur with random messages and can occur at any point during the mass copy.
Looking at the source mbox file in Local Folders, I can see that the message that fails actually has a length of 1002 bytes. So somehow tb is determining the wrong length for the email before sending it but sends the correct number of bytes.
It may also be possible for tb to tell the server that the length of the message to be appended is too large. This will cause the imap connection to timeout since now the server is still waiting for bytes that are never sent. I haven't actually seen this occur yet.

Comment 91

[jdien07]

I'm also seeing some corruption of messages. For example, the date of one message went from 2000 and 1935 during a copy by TB. I'm really glad you're tracking down these issues! I think I'll hold off switching to TB until you've had a chance to fix them. Let me know if there is anything else I can do to help, like beta-testing.

Comment 92

[gene smith]

(In reply to gene smith from comment #90)

Testing with jdien07's mbox file he sent me with about 17k messages, when doing a mass copy from Local Folders to a gmail imap folder, something seemingly random occurs. Each message is "appended" to the gmail mailbox and with the initial append request sent by tb there is a length, e.g., {1000} indicating 1000 byte are to be send by tb. Tb starts sending and sends 1000 bytes correctly but then 2 more bytes are sent. This causes gmail to respond with an error and the copy stops.

Ok, I was wrong about this. Actually, when the message length is X bytes, X bytes are sent but the append requires that two more byte be sent, CRLF to "terminate" the append. This is illustrated in the append example in rfc 3501 but you have to count the bytes to see the the two extra CRLF bytes are transmitted and are not part of the message byte count in {}'s.

However, gmail does respond with an error saying "BAD Invalid Arguments: Unable to parse message" but there is no apparent error in the data sent. (Of course, I can't really see the data the gmail receives.) I can retry the append and the exact same data is sent and gmail is happy. I have added a retry to the tb code to handle this error and do the retry. With this in the code I have been able to upload the complete mailing list to gmail (17k message some with fairly big attachment) several times with no error. (It is currently just test code and a formal patch has not been submitted to tb project.)

I have tested this with two other servers (my isp on openwave imap server and a dovecot server on localhost) but never see the same error that gmail reports. I have copied the same 17k messages twice to openwave and once to dovecot with 0 errors.

Tests so far have been from Local Folders to imap folders on gmail, openwave and (localhost) dovecot. I need to test to other servers now and then copy between imap servers. I am monitoring the transfer with wireshark and recording IMAP:5 log with tb to see where the errors occur.

This seems to occur with random messages and can occur at any point during the mass copy.

This is true, at least for gmail. Haven't seen it with openwave or localhost dovecot but need to check some other servers.

Looking at the source mbox file in Local Folders, I can see that the message that fails actually has a length of 1002 bytes. So somehow tb is determining the wrong length for the email before sending it but sends the correct number of bytes.

Checking again, tb is sending the correct message length and then sending the CRLF to terminate the append. So tb is doing the right thing.

It may also be possible for tb to tell the server that the length of the message to be appended is too large. This will cause the imap connection to timeout since now the server is still waiting for bytes that are never sent. I haven't actually seen this occur yet.

Still TBD on this.

(In reply to jdien07 from comment #91)

I'm also seeing some corruption of messages. For example, the date of one message went from 2000 and 1935 during a copy by TB. I'm really glad you're tracking down these issues! I think I'll hold off switching to TB until you've had a chance to fix them. Let me know if there is anything else I can do to help, like beta-testing.

On each append, tb sends the date that the message should have. Not sure why that is necessary since the date is already in the header that gets sent. Not sure why 2000 would be changed to 1935 but maybe 1935 is the start of the "unix epoch" or something like that. I'll check on this.

Comment 93

[jdien07]

once you're ready with the new code, I'd be happy to beta test with iCloud and do some careful testing and report back on what I see.

Comment 94

[Comfine GmbH]

gene, thank you very much for taking care of this issue!

we had this problem with our mailserver in the past, this is courier/exim. if it helps i can create a mailbox for you and give you the credentials for testing.

Comment 95

[gene smith]

Comfine GmbH,
Thanks a test account on a real server is usually a good way to see a problem. So yes, that would be fine. You can send me creds via email to my bugzilla profile address, if that's OK.
Can you also describe the copy/move scenario for you that doesn't work or often fails.

Comment 96

[Comfine GmbH]

gene smith,
i sent you Login-Data to your mail. The problems happend for me personal when i was moving a large ammount of mails from a folder of one account to a folder in another account. the move-process just stopped somewhere without response but not all mails have been moved - the source-folder was not empty. so i selected "the remaining mails" and started again to movie until i realized after the second time the process just aborted, that the mails at the destination were there two times while some was missing and some was remaining at the old folder, so i cleaned up that chaos and moved just small ammount of mails which worked fine than.
this is years ago but i noticed this behavier from time to time while transfer mails for customers (different providers/servers) also. hope this helps.

Comment 97

[gene smith]

Thanks Comfine G. for the test account. I will use it in my tests real soon.

Comment 98

[gene smith]

After testing copying from Local Folders to an imap folder I noticed this behavior: see bug 1579341 comment 31.
No one answered the question, but I determined the reason on my own. The full message fetches occur due to enabling "Enable adaptive junk mail controls for this account". This causes the most recent message bodies, on initial sync, to be scanned to detect spam. With this "Enable adaptive..." disabled, only a selected subset of message headers (From, To, Date etc.) are fetched for all message on initial sync with no offline storage and full message fetch does not occur until a message is opened.

Comment 99

[gene smith]

Comfine G, I was able to copy jdien07's 17k message mailing list to your server from Local Folder. Ran overnight with no errors. Now copying it to from your server to my ISPs server. Going ok so far. Monitoring with wireshark and TBs IMAP:5 log to detect any errors or stoppages.

Interesting factoid: It seems that when I try to copy to yahoo imap or aol (both use the same server and some ISPs use yahoo too) I always get an error on the append indicating [SERVERBUG]. However, this seems to be somewhat variable in that sometimes it works but only rarely for me when I copy only a single message. So mass copy to yahoo based servers (uses imap append) seems almost impossible.

Comment 100

[gene smith]

Copy to my ISP server from comfine server worked with no errors.

Another interesting factoid: My ISP's server seems unable to fetch the requested header items and returns empty. However, all message are present and completely accessible when opened, just subject, correspondent etc are blank on the message list. I think this is because the headers from jdien07's mailing list are too complex or maybe too long for my ISP's server (Openwave) to parse.

Now copying back the 17k messages to comfine server from ISP's Openwave is in progress...

Well, didn't go so well:

1. 3260 messages copied OK. Message ID 3261 fetched from Openwave OK.
2. Tb appended message ID 3261 at destination comfine server. Good.
3. Tb fetched next message ID 3262 from openwave, got only part of 226550 bytes message and
4. Openwave issued a tcp FIN causing a disconnect. Don't know why.
5. Tb immediately opened a new connection and fetches flags as expected.
6. Tb then fully fetched message 3262, all 226550 bytes using the new connection.
7. No append to comfine occurs as it should, so copy stops.
8. Last progress logged "CopyNextStreamMessage: Copying 3261 of 17257
9. Connection to comfine times out after a while due to inactivity as expected.

So this is an example of how a copy starts, runs for a while and then just stops. No errors are logged or notified to the user.

However, I'm not sure this is what the typical commenter to this bug is doing. For one thing, I am running the openwave and comfine accounts without offline store. So copy between them has to fetch the messages from the source server before appending to the destination server. When the bulk copy is from Local Folder or from an imap folder with offline store, the message to be appended is obtained directly from file and no fetch on the network of the message typically occurs. So having offline store should eliminate the error seen above, and I've not seen this when the source folder is in Local Folders.

However, the copy process should be able to fully recover from the above problem. It currently partially recovers by re-fetching the incomplete (due to openwave disconnect) message; but does not continue on and append the correctly fetched message to the destination server. The question is why not?

Edit: Tried to resume the copy by selecting message 3261 to the end on openwave (source) server and copy to comfile (destination). Copy would not start and nothing logged. Tried twice. Had to restart tb to resume the copy. I think this was mention by others above. Apparently a variable is set and locking out further bulk copies. More to look for...

Comment 101

[gene smith]

I have some more information to add to the scenario described in comment 100. When tb fetches the message to be copied from the source server (Openwave) it streams the received message to a temporary file. At step 4 when openwave issues the FIN, the temp file contains the partial message. However, even though tb retries the fetch and successfully receives the whole message at step 6, the data never gets re-streamed into the temp file (the partial temp file is not deleted and a new temp file is not created). I think what is happening is that the temp file that was partially written remains open and the fetch retry essentially streams to /dev/null (bit bucket). Also, there is no completion signal that would cause the partial temp file to be streamed out (read) and imap appended to the destination server (comfine) and then deleted. Instead the whole thing hangs and requires a tb restart to allow additional copies from source to destination servers. Testing this multiple times, I see multiple nscpmsg* files accumulating in the /tmp directory.

I haven't been able to find a solution for this so any suggestions would be appreciated. Also, this may be not be the typical copy/move scenario described by commenters and reporters to this bug.

Comment 102

[gene smith]

One other thing noticed after dragging a folder from Local Folders to root level of an imap account: If the copied folder in the imap account is deleted and then the drag/drop is done again, nothing happens. And even if a different folder in Local Folders is dragged, the copy doesn't occur. If the drag is to an existing folder in the same destination imap folder (destination not root level), the copy occurs OK. To drag again from Local Folder to that imap account root level requires a tb restart.
Note: Dragging between accounts only does a copy. Move between accounts (Local Folders is an "account") never occurs.

Comment 105

[David Hale]

Is this ancient bug any closer to getting fixed?

I'm still seeing this problem regularly.

A common job I have to do is migrate people off of Windows Live Mail.
So I set them up with Thunderbird, import their stored mail into "Local Folders" (if not all on IMAP) using ImportExportTools (note: it's dissappointing that Thunderbird has so few built in import tools), then drag/drop or Copy the folders to the IMAP structure.
This is very very flaky and often has to be repeated many times, failing at different points with no error message. Whereas it should simply be a case of doing the drag/drop or copy/move and walk away and leave it to sync.

A "copy/move without allowing duplicates" would also be good.

Comment 106

[gene smith]

Is this ancient bug any closer to getting fixed?

I've been working on this bug off and on since last summer. I didn't see any problem when moving/copying from local folder to imap folders, but sounds like you do. I do see problems copying between folders on different imap servers, but only when the source server setup in tb has no offline store and the source server decides to disconnect.

I have a local folder with 17k messages, some very large, that I have copied to several imap servers with no problems. So I didn't think this is a problem. Can you describe in more detail what you see, like how many messages, copying a single folder or tree of folders, drag/drop or using the menu, etc.

A "copy/move without allowing duplicates" would also be good.

Unfortunately preventing duplicates and/or restarting the copy where the failure left off would be very hard to put into the existing tb code. So I have mainly been trying to make the existing copy activity respond to errors better rather than just stopping like it sometimes does now.

Comment 107

[jdien07]

I imagine an important part of the equation is which IMAP server. I have problems quite reliably and the server is Apple's iCloud service. Gene, have you had a chance to try out that one yet? It's a free service so you could easily set one up for testing purposes if you haven't already. David, what IMAP server are you using?

Comment 108

[David Hale]

I'm doing this migration for a customer.
Their email is sky.com which is Yahoo based.

Thunderbird syncs up OK with the Sky/Yahoo IMAP server.
I am able to import WLM mail to Local Folders (Using ImportExportTools)
But when I drag/drop or Copy mail from Local Folders to the IMAP structure, it starts to work but then eventually stops randomly, giving me an incomplete copy. No error message. Fails at different point most times.

I have tried copying single folder and a tree of folders.
One folder I'm having problems with only has 1700 messages, which isn't that high.

Related question:-
I've always thought that for Thunderbird to succeed, importing mail from other mail clients should be much easier.
I'm a tech so using ImportExportTools isn't beyond me, but would it be possible to have good import filters built in to Thunderbird (that work) for common email clients like WLM. The need to import from WLM must be more common now that MS have retired it.

Comment 109

[gene smith]

(In reply to jdien07 from comment #107)

I imagine an important part of the equation is which IMAP server. I have problems quite reliably and the server is Apple's iCloud service. Gene, have you had a chance to try out that one yet? It's a free service so you could easily set one up for testing purposes if you haven't already. David, what IMAP server are you using?

Thanks for the tip. I have an iCloud account but I've only used it on a mostly dormant 2008 MacBook Air for simple tb tests, not bulk copies. Do you see problems copying to or from the iCloud account? Does it involve Local Folders too?

Comment 110

[gene smith]

(In reply to David Hale from comment #108)

I'm doing this migration for a customer.
Their email is sky.com which is Yahoo based.

I think I tried some bulk copies from Local Folders to yahoo and worked OK. Will try again. Also I think AOL uses yahoo server, FWIW.

I have tried copying single folder and a tree of folders.
One folder I'm having problems with only has 1700 messages, which isn't that high.

Guess if each message has megabytes of attachments it could be pretty big. With normal messages, yeah, not that big. This is also from local folders to the yahoo based server?

Related question:-
I've always thought that for Thunderbird to succeed, importing mail from other mail clients should be much easier.
I'm a tech so using ImportExportTools isn't beyond me, but would it be possible to have good import filters built in to Thunderbird (that work) for common email clients like WLM. The need to import from WLM must be more common now that MS have retired it.

Good point but maybe out of the scope of this bug. If doesn't already exist for this, maybe an "enhancement" bug should be entered. I don't know much about "wlm" users. Are they on a non-imap server like exchange?

Comment 111

[David Hale]

Yes my copes were from Local Folders to the Yahoo based IMAP server.

I have now finally got all the mail uploaded but it was quite painful.
I got lots of failures but when I re-tried and completed it last night it went without a hitch. Makes me wonder if it was a gitch on the IMAP server that got solved? I still think it's a general problem though as I have had these problems many times when doing migrations from other mail clients where their mail wasn't all on IMAP. And if there are glitches on the IMAP server, Thunderbird need to be able to handle it or at least give an indication.

WLM is just a mail client, so the user could be hooked up to any mail service over either POP or IMAP.
The only way I've found so far to import the mail that's not on IMAP is to use ImportExportTools.

Comment 112

[Barryg]

AT&T just forced me to revise Thunderbird from POP mail servers to IMAP mail servers in order to continue using my 20 year old bellsouth.net email addresses. Creating the new accounts in previously trouble free Thunderbird was not a major issue but now I am experiencing seemingly unresolvable issues as discussed above transferring the mail folders from the old POP to the IMAP accounts. My inbox has almost 9000 emails and there are about 20 sub-folders under the Inbox for specific email categories and the number of files in each vary from a few hundred to few thousand. I also installed the 64 bit version of Thunderbird (68.3.0) but that solved nothing.

Using Windows Explorer, I tried finding the actual file folders containing my Thunderbird emails history but could not locate them or at least find anything I could identify. Is it possible to work around this problem by simply copying the appropriate folders using Windows Explorer to the appropriate new file location? I would think with 32 GB RAM and all the horsepower my CPU provides this machine should not choke if I simply do file copy/pastes using Windows explorer. Any suggestions on how to locate and identify these folders would be appreciated. I am sure they exist somewhere as I regularly use Mozbackup and it obviously knows how to find them and where to put them if it is used to restore.

Comment 113

[gene smith]

I don't think you can or should try to fix it by just copying the pop storage files to the imap storage location on tb. FWIW, the pop files should be somewhere like under c:\users\<your name>\application data\mobile data\thunderbird\<your profile>\mail. I'm not on windows now so don't remember the exact path; you might also see it in tb by looking under your account's server settings near the bottom which works for imap, not sure about pop.

I assume you want to duplicate your pop folder structure under the new imap account. Also, I assume the imap account is working ok and can receive emails into Inbox. So have you tried to copy a single top level folder from the pop account to the new imap account? When you do this, what happens? This should cause the copied pop folder to be copied, mail by mail, to the new folder on the imap server. Note: To copy a folder you must drag it to the imap account. Before dropping, hold down ctrl key and see a small '+' to ensure it is copied and not moved (using ctrl key probably not needed since between accounts drag/drop always just copies, but suggesting this just to be careful).

Comment 114

[Barryg]

Wow! Thanks for the quick response. It is certainly appreciated.

After writing the message above out of frustration I looked again and in a path similar to what you suggested I did find both the IMAP and POP accounts and did find files named "Inbox" and "Inbox.msf" in both accounts (except "inbox was all in caps in the IMAP folder). There were also "Sent" and "Sent.msf" files. The very large size of the files without the .msf extension were obviously the files with all the emails. However, although tempted, I am hesitant to just copy them from the POP to the IMAP folder as I am not sure of the function of the .msf files and if they need to be copied too. Be curious about your thoughts of doing this.

All your assumptions are exactly correct. With respect to the Inbox it seemed I had to copy the contents from the POP inbox to the IMAP inbox and this does not work as described at the beginning of this thread. I also dragged some of the sub folders with varying degrees of success but not so successful that it is proving viable. In all cases the email or folder transfer just stops. Sometimes most of the emails get transferred, sometimes only a small percentage, but never all of them. Very frustrating for something I was not expecting any problem with.

Its obvious from reading all the posts above you have extended very significant effort to solve this problem and I need to tell you how much that is appreciated. I have been using Thunderbird ever since MS discontinued Outlook Express and this is the first Thunderbird has let me down so I am hoping to resolve this problem and continue managing my email as usual.

Comment 115

[gene smith]

Yes, copying the files and .msf would not work. Tb needs to see the messages on the imap server and fetch them from the imap server and then create its own files in the imap directory based on what it obtains from the imap server. The act of copying from the pop folder to the imap account/folder actually just sends the messages to the imap server using a command called imap "append". Then when the messages are all copied (appended) to the server's folder, tb fetches them and creates the storage files in the imap directory.

Let just try to get Inbox fixed before moving to other folders. How many messages are in Inbox? It sounds like maybe the server is dropping the connection and refusing even the automatic retry that tb does. You might consider recording a tb IMAP:5 log while doing a copy to know what is really happening: https://wiki.mozilla.org/MailNews:Logging . Then attach the log using the "Attach file" link above.

Another thing to try is add a column to pop and imap inbox, using the widget at the far right, called "order received" and sort on it. When doing the copy, tb starts with the lowest number and works up. This way you can more easily tell how far the copy went before the server refused to cooperate and only copy from that point when you try again to avoid duplicates. I realize this isn't optimal but right now that's all there is. Otherwise you will probably need to delete all the messages copied so far in Inbox and copy Pop/inbox again and hope they all get copied before the server craps out to avoid dupes.

Comment 116

[Magnus Melin [:mkmelin]]

For such cases, just move the message files to under the Local Folders, then inside thunderbird UI, move them to the IMAP server.

Comment 117

[gene smith]

Any non-Inbox folders he has created in the pop account are basically the same as "Local Folders" so moving them explicitly to Local Folders before online copy (append) to imap wouldn't be any different (at least that's what I assume). Comment 111 and others report problems copying from specifically Local Folders to imap folder.

Comment 118

[David Hale]

Sounds like the recent poster has had the same problems I reported recently when trying to copy large numbers of email to an IMAP account.

You may be right in that's it may not be a Thunderbird bug as such, it could be the IMAP server dropping connection during the copy of a large number of messages. But could there be a way to make Thunderbird handle such server connection drops better?

Rather than simply failing, perhaps come up with a dialogue box stating that the server dropped connection, and allowing you to retry from where it got up to, or quit? Possibly also a tick box allowing it to "keep retrying".

Comment 119

[Mike H]

I just encountered this problem as well. Could someone maybe update the documentation to tell people not to do this?

https://support.mozilla.org/en-US/kb/switch-pop-imap-account

Thanks.

Comment 120

[José Ángel Morente]

(In reply to David Hale from comment #118)

Sounds like the recent poster has had the same problems I reported recently when trying to copy large numbers of email to an IMAP account.

You may be right in that's it may not be a Thunderbird bug as such, it could be the IMAP server dropping connection during the copy of a large number of messages. But could there be a way to make Thunderbird handle such server connection drops better?

Rather than simply failing, perhaps come up with a dialogue box stating that the server dropped connection, and allowing you to retry from where it got up to, or quit? Possibly also a tick box allowing it to "keep retrying".

Exactly. You put the finger on the sore spot.
The problem is not that connection sometimes breaks; that may happen, most of the time due to network or server failures. But Thunderbird MUST be able to deal with it by: a) informing the user in a proper way and b) having the ability of resuming the broken operation. None of these issues are currently solved in my opinion. I also think it should be a basic feature and can't understand why it isn't properly implemented after 16 years of development (!!).

Blaming the IMAP server for this is a cheap excuse. The problem is a blatant poor design of Thunderbird.

Comment 121

[Mike H]

It's hard to tell if I was successfully able to swap emails between Local Folders and IMAP. How do I get a count of read and unread emails in each location?

Comment 122

[gene smith]

(In reply to Mike H from comment #119)

I just encountered this problem as well. Could someone maybe update the documentation to tell people not to do this?

https://support.mozilla.org/en-US/kb/switch-pop-imap-account

You mean don't use imap or don't try to copy from pop to imap?

(In reply to Mike H from comment #121)

It's hard to tell if I was successfully able to swap emails between Local Folders and IMAP. How do I get a count of read and unread emails in each location?

Not sure what you are asking. When a folder is selected, be it imap, pop or local, in the status along the bottom there is an "unread" and "total" count displayed.

Anyhow, how many messages and/or folders were you moving? Was it between pop or "local folders" to a new imap account? Any other details you can provide?

(In reply to José Ángel Morente from comment #120)

Blaming the IMAP server for this is a cheap excuse. The problem is a blatant poor design of Thunderbird.

I've never said it is entirely the server's fault. But otherwise I agree. There are no hooks I can find that notify the user when a problem occurs and no way to restart from where the last copy failed. Tb just stops. When bulk copies/moves are occurring internally tb really doesn't know this is going on and is busy doing lots of other things. What it needs is a dedicated bulk copy/move mode and UI where that is all that is happening, similar to the dedicated program imapsync.

Comment 123

[dennis]

I had the same problem. Tried to move 17k messages from IMAP inbox to an 'archive2019' called IMAP folder. Got into an endless loop (tb tried to move messages which have been already moved) as soon as I hit the inbox after restart of tb. What did stopped tb from trying to move the messages was to repair the inbox - then it suddenly stopped trying to move already moved messages.

Comment 124

[u654454]

I've read through the thread and I am also experiencing the issue described.

My usage scenario: I've been using TB since ~2011, with a < 1GB IMAP limit until recently so I've been creating annual archive folders to keep within IMAP quota. My company recently moved to Outlook 365 with 100GB storage and I also received a new work PC. As such I want to move all my offline emails archived in TB back to my inbox so I can use Outlook or a new install of TB on another PC as well as access all my email online. My annual archives range from 3,000-7,000 emails each.

My steps: Go to an archive folder, e.g. "2016", select all emails, right click -> Move To -> [e-mail] -> Inbox.

Observed results: Status bar shows "Copying message x of y to Inbox..." with green status bar. x starts at 1 and increments towards y (y being 3k-7k in my case). At some point before they're all copied (i.e. before x == y), the copying stops with no error message. Some but not all of the messages are copied. I don't know how many exactly copied, but I've seen the status bar showing close to 1,000 done at times. None of the source messages that were copied are deleted if x didn't reach y.

If I try this with a small number of messages, say 1-100, the process completes successfully, i.e. the messages are all copied, and the source copies are deleted.

This all seems to be inline with the previous comments so I'm not looking for further diagnosis of my failure.

One recovery proposed above is to sort source/destination folders by received date and compare to see which emails copied successfully, delete them in source, and then restart. I don't think that's feasible for me. I'll likely need 5-10 attempts for each archive folder, times 8 years, times however long it take to parse after each failure which adds up.

What I am wondering:

There is some discussion/ description of this move process (copy all selected messages, then delete source) being called bulk move/copy. I assume from this that a different process where you copy one message at a time, then delete its source, then copy the next message, etc, could be referred to as non-bulk move.

Based on this thread, this bug when bulk moving many messages has existed for at least 10 years with no resolution. As such, is it possible to disable bulk move and put TB into a mode where when you select multiple messages and tell it to move them, it would do each one separately as a non-bulk move? I.e. copy msg 1, delete its source, copy msg 2, etc,? If this is not currently an option, could that be added as a feature while the designers continue working on fixing real bulk move?

That would at least make it easier to continue the move process after a failure since you wouldn't need to parse through your source and destination folders comparing email dates, then deleting the source emails that have already been copied. This could be an advanced, non-default option so that the majority of users moving small numbers of emails would continue to get whatever advantages bulk move offers.

Thoughts?

Thanks.

Comment 125

[gene smith]

Hi spsilk,

My steps: Go to an archive folder, e.g. "2016", select all emails, right click -> Move To -> [e-mail] -> Inbox.

Are you moving between different imap accounts here or withing the same account. The reason I ask is because when the move/copy is intra-server, the sever does most of the work but when inter-server, tb must send all the messages to the other server via imap append.

There's not really a "bulk" mode vs. single message move/copy mode. When tb move/copies messages to another server it always just appends each message regardless of how many.

Also, move doesn't always delete the messages at the source. I think when you drag a folder to another server, tb always leaves the source folder in place. However, intra-server d/d does remove the source folder. (But I'm not 100% sure on this.) Anyhow, nothing is ever deleted if the move fails or hangs, I'm sure of that.

One recovery proposed above is to sort source/destination folders by received date and compare to see which emails copied successfully, delete them in source, and then restart.

My suggestion was to sort by "Order received" not necessarily by date. This orders the messages by imap UID and tb always copies from lowest to highest UID. (But maybe someone suggested order by date in the many comments above. :))

Anyhow, thanks for the inputs.

Comment 126

[u654454]

(In reply to gene smith from comment #125)
Hi Gene,
Thanks for the reply.

Are you moving between different imap accounts here or withing the same account.

I am moving from TB's "Local Folders", i.e. locally stored files. So, not moving between two IMAP accounts. To be clear, these are emails that were initially in my IMAP account, and were moved offline via TB's Archive action.

There's not really a "bulk" mode vs. single message move/copy mode. When tb move/copies messages to another server it always just appends each message regardless of how many.

OK, I'm not familiar with the inner workings of email protocols, but what I've been thinking of as copying, you're referring to as appending (to the IMAP Inbox?). In that case, my question still stands: couldn't there be a setting that makes TB append on at a time, deleting the source after each append? Rather than appending all the selected emails before beginning deletion, wherein you run the risk of the silently erroring out during the bulk append, leaving the source emails intact.

Thanks.

Comment 127

[gene smith]

I've discovered an interesting thing when copying message from Local Folders to an imap server. When the message is large and/or the server is slow, tb sends all the data via imap append (in multiple segments) and waits for the final OK response from the imap server. If it takes more than 20 seconds for the OK to be received, the transaction times out, tb drops the connection and does a retry, which also fails and the copy stops. Here's where the timeout is set:
https://searchfox.org/comm-central/rev/6f8371b5697c7e34a8a594a5f449e1e51f68fe35/mailnews/imap/src/nsImapProtocol.cpp#2124
When copying large messages to outlook.com, which is slow, I notice that sometimes it takes 60 seconds or more for the final OK to be sent by outlook.com. So if I change the code to allow appends to timeout after the default of 120 seconds, I can successfully copy messages to outlook.com that quickly timed out before. (I'm not sure why the code implies that append response should almost be instantaneous.)

Another thing is there is an add-on called "copy folder" that is referred to in comment 26 above. I tried it and it and since it also uses the same imap code, the append timeout remains 20 sec so it also fails for large messages to outlook, but it keeps going and copies subsequent messages instead of just stopping. It does 5 retries instead of the default 2 and appears to have the ability to detect existing messages in the destination folder and not duplicate them when the copy occurs again. However, even the latest version called "copy folder mod" only works up to version 60 and not with 68. See https://addons.thunderbird.net/en-US/thunderbird/addon/copy-folder-mod/?src=ss . I haven't studied how it works internally but I'll take a look and see if maybe it's functionality can be incorporated into mainline tb.

Comment 128

[Todd]

I have a YUGE migration to g-suite in the works. Since g-suite will not migrate over folders, but only the inbox, I have to move the customers folders from backup into their local folder and then restore them by copying them over. Because of this bug, I will be running my customer bill up through the roof, not to mention testing my own sanity. PLEASE FIX THIS!!!!

Comment 129

[gene smith]

Todd, are you actually seeing the problem where the copy from Local Folders stops mid-way when copy to g-suite? If so, how many messages in source folder and their approx. average size?
I haven't test this with Local Folders --> gmail recently to see how it behaves. (I'm assuming gmail and g-suite would behave in a similar way.)

Comment 130

[Todd]

I am going to have to do this shortly with imap.gmail.com. I have recently been put through this bug with exchange.charter-business.net. About drove me insane.

Comment 131

[gene smith]

Can you describe more details of what problem you see. "This bug" describes several issues and not sure exactly which problem you had or are expecting to have in future projects.
FWIW, here I'm on charter but not charter-business. It does OK when I copy to it but messages with big headers when copied in don't show the headers in tb. All other servers display the messages OK in tb so doesn't seem to be a tb problem. Charter (aka, Spectrum) uses openwave imap server, don't know about charter-business.

Except for the short timeout of 20 seconds described in comment 127 which is still present, copy from tb local mbox files should be fairly reliable. A big problem occurs when tb has to first fetch the messages from the source server 'A' before they are sent to server 'B' using IMAP append (see around comment 100). Therefore, it is best to first make sure the source imap folder has a local copy of the messages downloaded/sync'd before doing the copy. Of course, Local Folders and POP3 folders always have a local copy but optional with imap.

Comment 132

[Todd]

I am copying folders over from local to an account on g-suite now. time out like crazy! Attachment cause it to barf

Comment 133

[Todd]

I am going to have to do this shortly with imap.gmail.com. I have recently been put through this bug with exchange.charter-business.net. About drove me insane.

Okay, anything with an attachment oer 3.0 MB will reproduce this issue

Comment 134

[gene smith]

Todd, I'm still not sure of the problem you are seeing, but assuming it is the timeout of the IMAP append when copying from Local Folders to an imap folder. I am right now building a modified ESR 68 build that increases the timeout described in comment 127. I don't know your computer type so I am building win32, win64, linux64 and osx64 at the "try" server site here:

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=38d97b3ca0e7c63771baf67893c1201a90a679c1

To obtain the executable, click on the green B next to the appropriate architecture. Then when the options appear after clicking the green B, select the "Job Details" column. You will see a long list of item most of which a not relevant. For windows you would use the "target.installer.exe" link or for linux you would use the "target.tar.bz2" link; osx uses "target.dmg".

It takes maybe an hour for the try build to complete so if you are working on this right now you will have to wait until you see a bold green B next to the computer type.

Comment 135

[Todd]

I changed mailnews.tcptimeout from 200 to 500 (500 of what, I do not know), and I am able to transfer 8 GB attachments now

Comment 136

[Todd]

I just got away with a 22 MB attachment

Comment 137

[gene smith]

Probably the time in ms that tb will wait for a tcp ack response; just a guess. I didn't see that in my test but at least it's adjustable unlike the imap append timeout that I set in the try build.
Edit: It is time in seconds and it is how log tb will wait for data to be sent by the imap server measured from when the last data was sent to the server by tb.

Comment 138

[Todd]

I am doing anohter large migration today. mailnews.tcptimeout helps a lot, but some messagea have to be moved by themself.

Comment 139

[Todd]

I notice just before the time out, I get a "sending logon information" note on the lower left

Comment 140

[gene smith]

(In reply to Todd from comment #139)

I notice just before the time out, I get a "sending logon information" note on the lower left

That would be expected if a connection to imap server is dropped. Tb then has to make a new connection to continue on and that requires logging on to the server.

I don't think you have mentioned the tb version you are using. If you are at >=68, Bug 1535969 added an imap keepalive setting in the config editor. I'm not sure it would affect the tcp timeouts you were maybe having.

Comment 141

[Todd]

Also, I have mailnews.tcptimeout set to 1000 now or 1000 seconds. It times out at about 15 seconds.

Thunderbird 68.4.2 64 bit for Windows

Comment 142

[Todd]

mail.imap.tcp_keepalive.enabled = true
mail.imap.tcp_keepalive.idle_time = 100
mail.imap.tcp_keepalive.retry_interval = 5

Should any of this be altered? And is idle_time milliseconds or seconds?

Comment 143

[gene smith]

I think it's OK as is. The idle time is seconds meaning if no tcp activity on the imap connection for 100s a keepalive packet will be send. If no response to the keepalive, tb will be resend on a 5 second interval. Not sure how many times it will resend it if no response. (I'll have to re-read Bug 1535969.)

Comment 144

[Todd]

Here is an interesting tid bit. The customer I am working on right now had 10 Mbps down and 2 Mbps up speed. I could not move anything larger that 5 MB. I asked the ISP to max me out: 30 Mbps down and 5 Mbps up, and now I am able to transfer 15 MB files, but not 20 MB files

Comment 145

[gene smith]

Todd, If you are copying emails from Local Folders or from another imap folder to a destination imap folder, you may be having a timeout waiting for the imap server to process the final response to the imap append. For big messages this can take a long time, maybe up to a minute for some servers. This can result in a timeout since the mailnew.tcptimeout gets set down to 20s for the append even if you have it set larger. So have you tried to tb version referenced in comment 134 above and see if it makes a difference? It keeps the timeout at your set value during the append.

Comment 146

[Todd]

comment 134 does not pick up my old profile and seems to be a stand alone program

Comment 147

[Todd]

had to mess with profile.ini and find the nightly exe on program files

Comment 148

[gene smith]

Another way to run it is to just download the target.zip (https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/THZZ2LLrR06H8qufQr8Wvw/runs/0/artifacts/public/build/target.zip), unzip it and with a cmd window, cd into the new directory and run "thunderbird" from the cmd line. Requires no change to already install tb.

Anyhow, if you ran with the "try" version, did it help at all?

Edit: I tried the above link on win10 and I had to run "thunderbird --ProfileManager" and select the profile. Otherwise it wanted to create a new profile. Then when I went back to run the default install of tb, I had to add the parameter --allow-downgrade to the shortcut. So maybe not as easy as I claimed.

Comment 149

[Todd]

with the 134 build, I an transfer lanrge attachment one at a time. If I try too many, it times out with no warning

Comment 150

[Todd]

after running 134 on your profile, you can not go back to 68.4.0 general release. I had to install a nightly.

Comment 151

[gene smith]

(In reply to Todd from comment #150)

after running 134 on your profile, you can not go back to 68.4.0 general release. I had to install a nightly.

Yes, that's why I had to add the -allow-downgrade parameter. This is an annoying "feature" that comes from mozilla code, from what I understand.

Comment 152

[Todd]

Hi Gene,

Did you recompile 134 with that switch? And if so, is it up for download yet?

Also, that was 68 ESR Proposed that told me 68.4.2 was a downgrade. Am I missing something?

-T

Comment 153

[gene smith]

The "try" build referenced in comment 134 has nothing to do with -allow-downgrade switch. It only prevents the mailnew.tcptimeout from being set by the code down to 20s during imap appends. With the "try" version the timeout remains at whatever you have it set in config editor.

What I mean is that if tb complains on startup about the profile being old and telling you to make a new one (which is not really true when you switch between my "try" build of comment 134 and a recent release build) you now have to start release tb with the -allow-downgrade switch, either at the command line or in the windows shortcut for the desktop icon, e.g.,

thunderbird -allow-downgrade

If you need more info or this or what I'm saying is not clear, feel free to send me email since this is a bit off-topic for this bug. Thanks.

Comment 154

[Todd]

Attachment: gary.png

I just got done troubleshooting a customer with all kinds of connection issues over his cell phone hot spot trying to send, trying to copy to his sent bin, and imap socksts errors. He is running 68.5. I used comment 135 and it also fixed the issue.

See the attachment for a list of his errors.

Chuckle, I had to use -allow-downgrade to get comment 134 to work.

Comment 155

[Todd]

comment 134. Stinkin' typos

Comment 156

[Todd]

Hi Gene,

I have used your #134 spin several times now on various customer sites. Thank you!

Do you know which general release your wonderful modifications will be included in? And any idea when that will be?

Many thanks,
-T

Comment 157

[gene smith]

(In reply to Todd from comment #156)

Hi Gene,

I have used your #134 spin several times now on various customer sites. Thank you!

Do you know which general release your wonderful modifications will be included in? And any idea when that will be?

Many thanks,
-T

Sorry, been meaning to get back to this. Glad to know it has helped you. I have submitted a separate bug for comment 134 (bug 1618455). I don't consider this a complete fix for the this bug (When attempting to copy or move large numbers of local messages to an IMAP server, it fails without any error or status message). I hope to look some more at it real soon.

Comment 158

[gene smith]

TB user Xavier contacted me via email and reported that the "try" build from comment 134 is no longer is available. I did the build again and he reported that his bulk copied worked better. Here's the info I provided to Xavier for the builds:

You can find a modified 68.7 build here:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&collapsedPushes=689680&revision=74bd64740e539f12f94b0bbfc77067f4583b5eff

You can also find a modified trunk build here:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=fbcb811f3cd8cc9e228919d2355cab89f7ce1fd6

I built it for all architectures since I didn't know which one(s) you need. Also, it appears that 68.7 is building OK. Note: I haven't tried to run any of these builds.

To obtain the executable, click on the bold green B next to the appropriate architecture. Then when the options appear after clicking the green B, select the "Job Details" column. You will see a long list of item most of which a not relevant. For windows you would use the "target.installer.exe" link or for linux you would use the "target.tar.bz2" link; osx uses "target.dmg".

Comment 159

[Ricardo M.]

I just came across this thread and by god was I surprised to see this is an ELEVEN YEAR OLD issue.

Truth being told I had ceased to use TB definitely, now due to a project where I need to keep an eye on 20 different inbox folders ... decided to go back at TB and give it a whirl.

I started by making a backup of two inboxes previously configured on TB (with approximately 200k email messages) onto a backup inbox we use just to archive old emails.

Moving/coping or whatever in TB is excruciating. It has been one of the most brutal experiences I've gone through lately. Doesn't really matter if it's gmail, as it also happened with an outlook/hotmail account.

Emails are not copied or moved, the process breaks half way, nothing more happens. We have to move the same emails again. Duplicate email finder seems to stop working as I've erased more than 3000 duplicate emails manually.

One of the worse things I've seen tho, are the "notifications" that only show two lines and you can't see the whole notification.
Not only you can't see the only notification, it seems there isn't a single place where you can access these notifications later, with the full text.

All these made me start looking around and quickly found this thread.
It's an eleven year old issue. ELEVEN. I mean, wtf. Goodbye again.

Comment 160

[jdien07]

This is a little unfair to Gene, who has been working diligently on this for the past year or so (not his fault the ten years before that). I am a little surprised this isn't getting more priority though. The fixes I'm seeing making it onto the release notes don't generally seem as urgent as this one. I would have thought that any bug that leads to data corruption and loss of e-mails would get more attention. Gene, how are decisions made about what to prioritize? Is there someone there that we could petition to make this more of a priority so you could focus on taking this over the finish line? At any rate, I'm also holding off on adopting Thunderbird for now. I'm not sure how much longer I can wait before I need to make a decision.

Joe

Comment 161

[Ricardo M.]

Thanks for your comment. I really don't want to blame Gene nor think he's the one to blame, on the contrary, kudos for him for picking this up and working diligently on it.
My rant is towards Mozilla Foundation or whomever has responsibility on this project.
There aren't many great email clients, there's definitely space for TB to claim a greater marketshare if it wasn't for the perceived abandonment of fixing serious bugs and more innovation on developing important features and the look and feel of TB.

According to the Email Client Marketshare on (emailclientshare.com) TB doesn't even show on the 10 list. I believe if this was cared for properly, could be on the top 5.

One bug it's driving me crazy and I've seen other people complaining about are the damn notifications (macOS) that don't show the whole error text and doesn't matter if you click on the notification or go through the activity manager, debuggers, anything, you can't see the errors anywhere. I can't even express how riled up it leaves me because it makes me feel like we're still in 2008.

But anyway sorry for using this topic to rant about all this, these two issues are killing me and I'm really disappointed with thunderbird.
All the best for Gene thanks for having this on your desk.

Comment 162

[gene smith]

Joe, Thanks for the mailing list archive you provided. It has lots of emails with big headers and many large attachments (brain scans!) that are a good test bed that I have copied a lot. Yes, and I do need to get back onto this.
A while back I updated this addon https://addons.thunderbird.net/en-US/thunderbird/addon/copy-folder/ for TB 68:
https://github.com/smitgd/addon-copyfolder-tb-esr68 . Fixing tb to do what this does is not trivial. Unfortunately TB's built-in move/copy stuff doesn't have robust retry or duplicate prevention and there is very little feedback if something goes wrong. My addon at github only works with 68. I've never worked with addons before but managed to port the <60 one to 68. I'm not sure what it will take to change it for the next release or current betas and daily. Anyhow, the last time I checked it, the ported addon worked very well with tb 68.
I can't speak for TB priorities but I think most effort goes into keeping tb current with the latest firefox/mozilla codebase and fixing or adding calendar functions. So this is looked at as somewhat as a niche bug since it doesn't affect most typical users. At least that's why it was basically ignored when reported about 11 years ago.

Comment 163

[gene smith]

Attachment: addon.xpi -- only works with TB ESR 68.*

I just tried my ported copy-folder addon with 68.8 and it still works. By default, it copies the folder recursively. But on the pref screen for the addon you can tell it to not copy subfolders but just the selected folder. There are a few other options that currently don't affect anything.
I've attached the addon.xpi file if anyone dare try it.

Note: This doesn't fix the tcptimeout problem that can occur with some servers (e.g., outlook.com) when the imap append response takes too long. But it will try a few more times which will also probably fail. It will then move on to the next message. The failures will be noted in the final summary. Then if you run the copy again, only the missing messages will be tried again.

Comment 164

[Ricardo M.]

Hi Gene thanks a lot for your message.

The add-on compatible with TB68 is the one at GitHub right? because this - https://addons.thunderbird.net/en-US/thunderbird/addon/copy-folder/ - says isn't compatible with my TB 68.8.1 so just to make sure. thanks

Comment 165

[gene smith]

The add-on compatible with TB68 is the one at GitHub right?

Right. The official copy-folder addon was never ported to 68. The one I put on github is ported to 68 and works only with 68.*. You don't have to "clone" or download anything from github unless you want to. You can just use the attachment "addon.xpi" from comment 163 and install it from file on the TB addon manager screen. (No warranty or guarantee!)
Once installed and tb restarted, right click the folder you want to copy, select the new (Addon) copy that will now be present and select the destination folder. Oh yes, before starting the copy, click on a folder at the destination sever to make sure you have a connection to that server. This is mentioned in the readme on github.

Comment 166

[Ricardo M.]

Alright I actually have moved all the big folders already but I'll keep this in mind.
I truly hope tho, that this makes it to TB code base instead of being an "add-on".

Thanks Gene!

Comment 167

[Peter Shute]

I'm being plagued by this problem at the moment. I've got over 18000 to copy to O365, and I just can't get consistent results. I'm going to try splitting it into folders of 1000, and copy whole folders at a time, so at least I can tell which ones failed.

I assume there are good reasons why this bug is still not fixed after 11 years. Perhaps a way forward would be to add some checks to the process so that users are at least warned there's been a failure. It would also be great if it was possible to identify which messages didn't get copied, so one could just try again.

Comment 168

[adam]

I am several thousand emails away from manually moving tiny batches of emails to gmail over and over again because of this bug and the only thing keeping me going is the thought of never using Thunderbird again. I can't believe this bug has existed for 11 years. Cool that people have decided to fix it in the last year but it is inexcusable for a piece of software to be this fundamentally broken for so long. Honestly pathetic.

Comment 169

[gene smith]

Attachment: 78-copyfolder.xpi -- only works with TB ESR 78.*

Here's the install file for my copyfolders add-on, updated for tb 78, described here: https://github.com/smitgd/addon-copyfolder-tb-esr68/tree/WL. Note: the version for TB 68 is on the "master" branch and the install file for 68 is attached above at comment 163.

Note: With 78, the add-on options are found under the Tools menu item. There is really only 1 option you can currently select: Recursive vs. non-recursive folder copy.

I would like to implement similar functionality in the mainline TB. Unlike current TB copy/move methods, this handles each copy separately so you can get somewhat finer-grained error indications.

If you are willing to try this, let me know if it seems helpful in doing the mass copies of whole folders.
Thanks and sorry for the extreme delay in getting this bug fixed.

P/S: copyfolder add-on attempts to preclude dupes in the destination folder after the copy. Another add-on tool that might be helpful is this: https://addons.thunderbird.net/en-US/thunderbird/addon/removedupes/?src=ss

Comment 170

[Peter Shute]

(In reply to gene smith from comment #169)

Gene Smith, have you done any comparisons to compare the speed of your one by one copy with the standard copy? Obviously it's going to work out faster than doing it over and over till it's right, but it would be good to know if it's going to run more slowly.

Comment 171

[showard314]

(In reply to gene smith from comment #134)

Todd, I'm still not sure of the problem you are seeing, but assuming it is the timeout of the IMAP append when copying from Local Folders to an imap folder. I am right now building a modified ESR 68 build that increases the timeout described in comment 127. I don't know your computer type so I am building win32, win64, linux64 and osx64 at the "try" server site here:

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=38d97b3ca0e7c63771baf67893c1201a90a679c1

In case others are finding this bug, I wanted to cross-post a comment and point to the other bug tracking it with an update.

This patch worked great! However, the change got backed out by a later commit and the treeherder artifacts expired. There isn't a way to test/use these patches at this time.
Info is on the other bug report:
https://bugzilla.mozilla.org/show_bug.cgi?id=1618455#c4

Comment 172

[gene smith]

(In reply to showard314 from comment #171)

See bug 1618455 comment 5 for my proposed patch. Sorry for the delay on this.

Comment 173

[csanad]

Had this bug as well (TB 78.7.0, Win 10) - tried moving thousands of mails from local folder to an IMAP (gmail) account, and the process always stopped (waay) before completion. Now trying extension 78-copyfolder.xpi from comment #169. Will report back if it works or not.

In the meanwhile, let me note that while it is running, status/progress is strange: it restarts counting sometimes after a few hundred, sometimes after 1-2-3 mails because of indexing happening.

Comment 174

[csanad]

(In reply to csanad from comment #173)

Had this bug as well (TB 78.7.0, Win 10) - tried moving thousands of mails from local folder to an IMAP (gmail) account, and the process always stopped (waay) before completion. Now trying extension 78-copyfolder.xpi from comment #169. Will report back if it works or not.

In the meanwhile, let me note that while it is running, status/progress is strange: it restarts counting sometimes after a few hundred, sometimes after 1-2-3 mails because of indexing happening.

Seems to work close to perfectly! Copied ~60k mails from a local folder to an IMAP (gmail) account on the first try. Apparently ~1% of the mails were duplicates, and there seems to be a loss of ~0.1% - or these 0.1% of mails could not be copied somehow in the first place, or the mail count within a folder (as given in the respective column next to the folder's name in TB) is only accurate up to such precision.

I am now trying another, ~80k folder, it also goes well so far.

Note that simple copying failed 100% of times (out of 10-15 attempts), so this is a HUGE improvement, thanks a lot!

Comment 175

[gene smith]

csanad,
Glad to here that the addon is helping. Yes, I noticed the counters resetting too but haven't had a chance to look at it yet. I've notice a few other problems too.
I would like to eventually port the addon to mainline tb. To me, it makes sense to right click a folder and be able to copy it one message at a time. With mainline tb, you have to drag the folder which seems to be basically undocumented and any error applies to the whole folder rather than the individual messages in the folder.
FYI, bug 1618455 is fixed and is now in tb daily and it may help with certain problems related to mass copies even when not using the addon.

Comment 176

[csanad]

Gene,
Thanks for your reply. Indeed this folder copy should be in the main TB version as well, as it is a missing feature otherwise, as you say; or even worse, the feature appears to be there but is quite buggy.
Also, now I tried it on another machine and somehow many times it just responds that X (sometimes X=0, sometimes many thousands) messages were copied, and the process was aborted. It would be good to somehow tell why it was aborted, to understand and be able to solve the issue.

Comment 177

[Wayne Mery (:wsmwk)]

Potentially related https://mzl.la/3skNjlz

Comment 178

[erich.legan]

This seems to be working wonderfully for me too! My one issue is that when I uploaded my inbox (it was a local folder) to my gmail imap server's inbox, it's now in a sub-folder of my gmail inbox.

Is it really easy to just copy the messages from this sub-folder since it's copying within the server and not from my computer? Or am I missing something with the recursive/non-recursive copy and should I be copying the individual messages differently somehow so I don't end up with a sub-folder on the server?

Comment 179

[gene smith]

I assume you mean by "this" the addon.

Anyhow, glad to hear it is working OK for you. Again, it still needs work and may not work correctly with future releases greater than 78 due to changes in how addons works. So it will need some adaptations.

The addon does currently just copy the folder name and all internal messages to the destination and not just the internal messages. So, yes, you can just copy the messages from the new subfolder of gmail's Inbox into inbox. Then when the copy finishes you can delete the subfolder. (You could also use the move function but, just to be safe, I'd do the copy and then delete the subfolder yourself when you are sure everything copied OK.)

Comment 180

[duccio.magini]

I'm using the addon 78-copyfolder.xpi with TB 78.11.00 (32bit) and it works without any stop. But it is very slow, like 1 email per minute.
Do I have to do the copy with TB in offline mode or there is any way to speed up the process? Is it so slow also for you?

Comment 181

[erich.legan]

The solution Gene gave worked perfectly for me. The way you have yours set up is exactly how mine was too but it went way faster than 1 email per minute. You might try emptying your gmail trash? I remember that caused some issues if the trash was too full—or at least I thought it did.

Also, you're highlighting just the folder and right-clicking on it to access the addon right? You're not highlighting all of your individual emails?

Last thing I can think of is that your internet speed is just really slow.

Comment 182

[timo.jyrinki]

Just to update the problem seems to still be there with Thunderbird 91.1, ie to give my case exactly pretty impossible to move large amounts of messages from local folders to IMAP (Google) since timeouts very often.

With the addon and using older Thunderbird 78 from a snap (since the addon seems to install but isn't functional on 91) I seem to be able to workaround the problem of moving back a folder with 4811 messages from local to IMAP (moved away to temporarily make up space on account), even though it's indeed very slow (maybe 10 messages per minute).

Comment 183

[gene smith]

(In reply to timo.jyrinki from comment #182)

Just to update the problem seems to still be there with Thunderbird 91.1, ie to give my case exactly pretty impossible to move large amounts of messages from local folders to IMAP (Google) since timeouts very often.

Must be somewhat network or location dependent. I've been working on another bug where I've copied more that 5G of messages to gmail Inbox with no timeouts using the built-in tb menu (not my addon).

With the addon and using older Thunderbird 78 from a snap (since the addon seems to install but isn't functional on 91) I seem to be able to workaround the problem of moving back a folder with 4811 messages from local to IMAP (moved away to temporarily make up space on account), even though it's indeed very slow (maybe 10 messages per minute).

I haven't attempted to update or even test my addon recently to see if it works with 91 or later. I'll try to get to it ASAP.

Comment 184

[John K.]

TB 91.2.0 64-bit on MacBook Pro just recently purchased - the problem is still there. I'm trying to migrate a user from one G-Suite based office to ours (also G-Suite). This user has 64,000+ messages in her Inbox and 92,000+ messages in her Sent folder that date back to Jan. 2015. It's imperative I get these emails copied to her new G-Suite account a.s.a.p. Both G-Suite accounts are IMAP so we can't "move" the emails - they have to be copied so we can leave the originals in the source account. The Copy-To option in TB only runs for about 5 minutes before it stops without any error message. Now when I try to pickup where it left off it won't copy any messages at all. PLEASE . . . is there an addon for this version to make this work. I'm running out of time.

Comment 185

[Markus Mobius]

(In reply to John K. from comment #184)

TB 91.2.0 64-bit on MacBook Pro just recently purchased - the problem is still there. I'm trying to migrate a user from one G-Suite based office to ours (also G-Suite). This user has 64,000+ messages in her Inbox and 92,000+ messages in her Sent folder that date back to Jan. 2015. It's imperative I get these emails copied to her new G-Suite account a.s.a.p. Both G-Suite accounts are IMAP so we can't "move" the emails - they have to be copied so we can leave the originals in the source account. The Copy-To option in TB only runs for about 5 minutes before it stops without any error message. Now when I try to pickup where it left off it won't copy any messages at all. PLEASE . . . is there an addon for this version to make this work. I'm running out of time.

If it's Google workspace to workspace migration why not just use the Migration service from Google? I have used it for exactly that purpose and it worked well.

https://support.google.com/a/answer/9476255

I never got the TB copying to work without errors (with moving it at least deletes the original so you know where to restart but i understand you want to just copy not delete).

Comment 186

[John K.]

Thanks Markus - I would have used the migration tool as well (have had success in the past using to migrate nearly 50 users in the past), but it was imperative that the admin of the source account/domain not be aware of the "move" (it's somewhat sensitive). Since I couldn't guarantee her current admin wouldn't know of the migration I went with the TB option instead.

Comment 187

[gene smith]

John K.
You might try my addon in comment 169 above. I haven't gotten back to adapt this to 91 so you will need to temporarily downgrade to 78. (You will need to run 78 wih --allow-downgrade command line option unless you make a new profile since you have already run 91 with the profile.)

Comment 188

[Comfine GmbH]

Gene, shouldn't your "addon" be integrated to the core? copy large ammounts of mails is just a basic-feature of a software. are you part of the development-team? if not, is someone of them reading here?

Comment 189

[John K.]

I have to admit that I'm somewhat of a novice admin here. That being said, I'm unaware of how one might implement a command line instruction on a Mac Powerbook Pro.
Minor update: the copy-to function seems to be working with small sets of of messages. Keeping to message count down to 250 at a time seems to be working just fine. Downside . . . I'll have to do this 625 times to get all her emails :-(

Comment 190

[gene smith]

(In reply to Comfine GmbH from comment #188)

Gene, shouldn't your "addon" be integrated to the core? copy large ammounts of mails is just a basic-feature of a software. are you part of the development-team?

I'm an unpaid volunteer and I mostly just look after the low level IMAP issues when they come up. Although bulk copy/move involves IMAP, it also reaches into to all the other protocols and involves a lot of UI stuff that I'm not a specialist on.

if not, is someone of them reading here?

If you look at the CC list above you will see mkmelin and wsmwk. They makes most of the decisions as to what has priority to be added or fixed in tb by paid staff. Apparently they don't deem this an issue that requires action (even though it has been a problem for at least 12 years with comments still coming in).

Also, thanks for letting me test with your server a while back!

Comment 191

[gene smith]

(In reply to John K. from comment #189)

I have to admit that I'm somewhat of a novice admin here. That being said, I'm unaware of how one might implement a command line instruction on a Mac Powerbook Pro.

I'm not sure either. I have an old "Air" that I haven't fired up in over a year with "Mavericks" OS. I suspect it's similar. I'll take a look and let you know how it's done.

Minor update: the copy-to function seems to be working with small sets of of messages. Keeping to message count down to 250 at a time seems to be working just fine. Downside . . . I'll have to do this 625 times to get all her emails :-(

Ugh...

Comment 192

[gene smith]

John K.
I'm assuming you know how to find and install the 78 version of tb on mac. Since you have already installed and run 91, if you try to run 78 you will be requested to create a new profile to continue. You can do that if you want but you will have to download all the messages in the accounts again. So to avoid this, you need to use the --allow-downgrade option on tb 78.

You can run the installed tb 78 by going to "Utilities" and running the "Terminal" app. In the terminal app you can run thunderbird like this:
open /Applications/Thunderbird.app --args --allow-downgrade.

There seems to be no "easy" way to add a command line option to the TB icon you normally click to run TB on macOS/OSX. So you have to run tb from a terminal.
There may be other ways but it involves writing "scripts" and running them as apps, but this seems to be the easiest for just a temporary use.
Re: https://superuser.com/questions/16750/how-can-i-run-an-application-with-command-line-arguments-in-mac-os

Edit: You can get the last 78 versions here (Assuming en-US locale): https://archive.mozilla.org/pub/thunderbird/releases/78.14.0/mac/en-US/
I installed from file my addon in comment 169 and ran it in terminal with "open /Applications/Thunderbird.app --args --allow-downgrade" and it worked OK with v78.14 when I copied a folder to an icloud account. Actually, I had never tried running the addon with macos until now.
FWIW, on this old macbook air with "mavericks" the os is too old and tb 91 won't install but 78 still does.

Comment 193

[Magnus Melin [:mkmelin]]

If someone can pinpoint what's wrong, it would be nice to fix. What does the add-on do differently that fixes it?

Comment 194

[gene smith]

Magnus, It's been a while but I think the main problem with tb built-in copy is that it sends all the imap UIDs in one big append chunk. So if it fails for some reason, you end up with just a partial copy to the destination and usually it just stops with no error or warning to the user. If you start it again, you get duplicates. A couple years ago I tried to understand and fix it but it was beyond me, so I found an old addon that had not been updated to new TB and adapted it.

With the addon, each individual message is handled atomically and you get an error message if there's a problem but it retries a few times before giving up on the single message copy. (I don't remember if it continues on to the next messages if one fails.) Then if you restart, it tries not to re-copy what has already been copied so it just fills in any missing messages.

Comment 195

[John K.]

Thank you everyone - the response here has been awesome - far better than I had hoped it would be. Gene - a special thanks to you and your continued contributions here! Being a novice and having so much time already invested in this, I'm going to stick with what I know is working. On a positive note, I'm up to copying about 500 emails at a time without a timeout/stall so I've just cut my number of copy groups in half :-)
Less than 300 groups of 500 to go. I run a group of 500 every 10 min.s or less (or about 50 in an 8-hour day shift). I'm sneaking these in between regular work so I should be able to put this in the done pile by the end of next week (sooner if I keep it going over the weekend).

Comment 196

[gene smith]

I've updated the addon to now work with 78, 91 and later (tested with 78, 91 and 97a daily). https://github.com/smitgd/addon-copyfolder-tb-esr68/tree/WL
This is branch WL (Window Listener). Only the "master" branch works with tb 68.

Comment 197

[Todd]

(In reply to gene smith from comment #196)

I've updated the addon to now work with 78, 91 and later (tested with 78, 91 and 97a daily). https://github.com/smitgd/addon-copyfolder-tb-esr68/tree/WL
This is branch WL (Window Listener). Only the "master" branch works with tb 68.

HI Gene,

Is there an XPI out there of your add-on somewhere?

Comment 198

[gene smith]

Edited comment:

Is there an XPI out there of your add-on somewhere?

I submitted it to the tb addon site and it still says "awaiting reveiw". Not sure what that means but this does seem to work:
https://addons.thunderbird.net/EN-US/thunderbird/addon/copyfolder-for-esr-78/

Submission was rejected since the addon uses legacy API and a "Window Listener Experiment" to adapt it. I didn't know that they require new addons to use "Mail Extension" API. So I just put the working XPI here for now: https://github.com/smitgd/addon-copyfolder-tb-esr68/releases/tag/v2.2.2

Comment 199

[jwtuttle]

Just wanted to note for all concerned that this behavior still exists in the latest version of Thunderbird, 102.7.1.

Same thing that everyone else has described over the past 10+ years. If you select a bunch of messages and try to move them to another IMAP folder, the operation will begin and some number of messages will be copied, but then the operation just... stops. There's no error message, no indication of what has been copied and what hasn't, and generally if you retry it a couple of times you will end up with a bunch of duplicates in the destination mailbox/folder.

The type of IMAP server on the far end does not seem to matter. I have seen it happen with Gmail's IMAP implementation and Dovecot running on my local server (across several versions of Dovecot at this point). I have also watched the Dovecot logs as I tried a big copy, and there is nothing in there to indicate an error on the destination side.

I can only imagine that more and more users will run into this issue, as (at least in my experience, and my friends/colleagues') more people have very large IMAP accounts today than 10+ years ago, and there are now people—myself included—who are running into the limits of free storage on services like GMail and are looking to migrate their messages, often many GB, out and onto other services/servers or self-hosted setups. Without the ability to do large copy operations, users are stuck using stuff like Google Takeout and then dealing with the resulting mbox files directly.

Comment 200

[ISHIKAWA, Chiaki]

Well, just to show how difficult it is to run tests using valgrind, here is an example of an incomplete test.
I reported Bug 1824691, it is about a runtime error when C-C TB is compiled using gcc-12 compiler suite.
I thought it was a simple miscompilation. (It could be.)
But then I suddenly recalled that there WAS a mysterious valgrind memory error reported in the hashtable implementation and it had something to do with string allocation some years ago.
And hashtable is in the stack in bug 1824691 and the error occurred int the string allocation.
So I thought there may exist uninitialized memory issue which may happen due to the particular manner the code is compiled.
It may happen sometimes and it may not show up sometimes depending on the manner the code is compiled/optimized.

So I tried to run the particular test under valgrind, but my attempt to test it failed even before the code hits the problematic assertion.
See below.
The point is

• the test involves a future activation of an event. It specifies 30 seconds for a future event, it seems.
  Usually, everything else finishes by then, and TB and the test framwork will wait for this event that happens at 30 seconds into the future.
  However, valgrind slows down the execution of involved program(s) to the point that the test framework still waits for "brower"(?) when 30 seconds passed. And indeed the test screen has an outline of the rectangle of TB window but only the frame lines only, nothing else, it is black empty rectangle with only the visible frame. That must be the so-called "browser" screen (in which, actually TB's content is displayed.)
  I wonder what would happen to the set alarm.
  Maybe when everything becomes ready under valgrind, the timer has already fired and TB cannot seriously wait for that event any more.
  Since nothing happens soon, I went to sleep.
  And this morning I found the whole test timed out.
• The runtests.py leaves room for improvement for timeout handling.
  I have hacked runtests.py to extend time out values here and there. But maybe the logical issue that the 30 second timer into the future in the original test may not be properly caught because the TB executes very slowly under valgrind needs to be addressed first.

So in order to run test(s) under valgrind in reasonably satisfied manner, various timeout values in TEST java code files themselves,
C/C++ source code, and runtests.py need to be adjusted. (I did it for |make mozmill| test framework of C-C TB by trials and errors several years ago, and it worked rather well for a few years. But new mochitest framework for C-C TB is very difficult to handle in this regard. I have run many of the xpcshell tests under valgrind, but for mochitests, I found too many tests timed out and so I could not get meaningful test coverage. This has been the state of affairs since C-C TB transitioned to mochitest framework a couple of years back. For mochitest, runtests.py seems to require much hack as far as C-C TB timeout issue is concerned. AND tests themeselves.
I tried address-sanitizer instead to check memory issues.
It can deal with out of bound type access issues, but cannot check for uninitialized value issues.

So unless someone who is fully employed to work on C-C TB steps in, I don't think any serious testing of C-C TB under valgrind would happen.
I don't even know if monthy run on tryserver for C-C under valgrind happens at all. I think it happens for FF, but I need someone familiar with FF development to learn more about it.

E.g., right now, I think the passing of valgrind arguments to valgrind on |mach| command line to run C-C TB under valgrind broke again [I found I cannot pass the arguments correctly any more last year], and so I had to use an outside proxy program that runs instead of C-C TB binary and adds desired valgrind arguments to an instance of valgrind that runs the original C-C TB binary with the originally passed arguments to C-C TB.
This passing worked two or three years ago for at least a few months if I recall correctly.

Oh well. Here it is the log snippet. I show the log about the scheduling of one-off event 37232 millisecond into the future happened.
This is for comm/calendar/test/browser/browser_calDAV_discovery.js which is one of the mochitest unit tests that experienced the bug 1824691.
Oh wait, I now notice another future event: " 10:25.73 GECKO(167286) [CalMetronome] Scheduling one-off event in 43341ms"

09:21.54 GECKO(167286) --167290-- failed to stat64/stat this file
09:30.86 GECKO(167286) --167290-- memcheck GC: 83263 nodes, 18636 survivors (22.4%)
09:31.82 GECKO(167286) console.debug: Calendar:
09:31.86 GECKO(167286)   [CalMetronome] Scheduling one-off event in 37232ms
09:49.45 GECKO(167286) --167290-- memcheck GC: 83263 nodes, 11726 survivors (14.1%)
09:58.40 GECKO(167286) [Parent 167290, Main Thread] WARNING: 'NS_FAILED(rv)', file /NEW-SSD/NREF-COMM-CENTRAL/mozilla/netwerk/base/nsSimpleURI.cpp:628
09:58.40 GECKO(167286) ==167290== WARNING: new redirection conflicts with existing -- ignoring it
09:58.40 GECKO(167286) --167290--     old: 0x04c04600 (__memcpy_chk_avx_una) R-> (2030.0) 0x048487d0 __memcpy_chk
09:58.40 GECKO(167286) --167290--     new: 0x04c04600 (__memcpy_chk_avx_una) R-> (2024.0) 0x04848170 __memmove_chk
09:58.42 GECKO(167286) {debug} SetSpec failed. : aSpec=messenger/otr/chat.ftl
10:04.89 GECKO(167286) --167290-- Reading syms from /usr/lib/x86_64-linux-gnu/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-svg.so
10:04.90 GECKO(167286) --167290--    object doesn't have a symbol table
10:04.92 GECKO(167286) --167290-- Reading syms from /usr/lib/x86_64-linux-gnu/librsvg-2.so.2.48.0
10:04.97 GECKO(167286) --167290--    object doesn't have a symbol table
10:12.99 GECKO(167286) --167290-- WARNING: Serious error when reading debug info
10:12.99 GECKO(167286) --167290-- When reading debug info from /memfd:mozilla-ipc (deleted):
10:12.99 GECKO(167286) --167290-- failed to stat64/stat this file
10:12.99 GECKO(167286) --167290-- WARNING: Serious error when reading debug info
10:12.99 GECKO(167286) --167290-- When reading debug info from /memfd:mozilla-ipc (deleted):
10:12.99 GECKO(167286) --167290-- failed to stat64/stat this file
10:25.71 GECKO(167286) console.debug: Calendar:
10:25.73 GECKO(167286)   [CalMetronome] Scheduling one-off event in 43341ms
10:39.84 GECKO(167286) --167290-- memcheck GC: 83263 nodes, 22867 survivors (27.5%)
10:41.23 GECKO(167286) [GLX] window 40006c has VisualID 0x50
10:42.14 GECKO(167286) GL_VENDOR: Mesa/X.org
10:42.14 GECKO(167286) mVendor: Unknown
10:42.14 GECKO(167286) GL_RENDERER: llvmpipe (LLVM 15.0.6, 256 bits)
10:42.14 GECKO(167286) mRenderer: Unknown
10:42.14 GECKO(167286) mIsMesa: 1
10:42.17 GECKO(167286) [Parent 167290, Renderer] WARNING: robust_buffer_access_behavior marked as unsupported: file /NEW-SSD/NREF-COMM-CENTRAL/mozilla/gfx/gl/GLContextFeatures.cpp:630
10:42.17 GECKO(167286) [Parent 167290, Renderer] WARNING: Robustness supported, strategy is not LOSE_CONTEXT_ON_RESET!: file /NEW-SSD/NREF-COMM-CENTRAL/mozilla/gfx/gl/GLContext.cpp:994
10:42.17 GECKO(167286) [Parent 167290, Renderer] WARNING: robustness marked as unsupported: file /NEW-SSD/NREF-COMM-CENTRAL/mozilla/gfx/gl/GLContextFeatures.cpp:630
10:42.57 GECKO(167286) [WARN  webrender::device::gl] Missing optimized shader source for gpu_cache_update
10:47.26 GECKO(167286) --167290-- WARNING: Serious error when reading debug info
10:47.26 GECKO(167286) --167290-- When reading debug info from /memfd:mozilla-ipc (deleted):
10:47.26 GECKO(167286) --167290-- failed to stat64/stat this file
10:47.61 GECKO(167286) --167290-- WARNING: Serious error when reading debug info
10:47.61 GECKO(167286) --167290-- When reading debug info from /memfd:mozilla-ipc (deleted):
10:47.61 GECKO(167286) --167290-- failed to stat64/stat this file
10:47.66 GECKO(167286) --167290-- WARNING: Serious error when reading debug info
10:47.66 GECKO(167286) --167290-- When reading debug info from /memfd:mozilla-ipc (deleted):
10:47.66 GECKO(167286) --167290-- failed to stat64/stat this file
10:47.67 GECKO(167286) --167290-- WARNING: Serious error when reading debug info
10:47.67 GECKO(167286) --167290-- When reading debug info from /memfd:mozilla-ipc (deleted):
10:47.67 GECKO(167286) --167290-- failed to stat64/stat this file
10:50.47 GECKO(167286) --167290-- REDIR: 0x491f5b0 (libstdc++.so.6:operator new(unsigned long, std::nothrow_t const&)) redirected to 0x483efd9 (operator new(unsigned long, std::nothrow_t const&))
13:13.37 INFO runtests.py | Waiting for browser...     <---- See, this is already close to 2:30 after the initial 30 second alarm was set.
                                                                                    or for that matter, the later  Scheduling one-off event in 43341ms.
14:33.35 GECKO(167286) --167290-- memcheck GC: 83263 nodes, 23239 survivors (27.9%)
_read(): after the while now 1680137194.8400612
_read(): after the while timed_out True
_read(): after the while timeout= 1680137194.831557
_read(): after the while output_timeout= 1680140464.303057
_read(): Timeout occurred, but should ignored for valgrind run?
_read(): stdout_reader= <Thread(ProcessReaderStdout-/NEW-SSD/moz-obj-dir/objdir-tb3/dist/bin/thunderbird, started daemon 139700052424384)>, stderr_reader=None
calling timeoutHandler(): timeout=  3700.0
calling timeoutHandler(): proc=  <mozprocess.processhandler.ProcessHandlerMixin object at 0x7f2eb849b1d0>
calling timeoutHandler(): utilityPath=  /NEW-SSD/moz-obj-dir/objdir-tb3/dist/bin
calling timeoutHandler(): debuggerInfo=  None
calling timeoutHandler(): browserProcessId=  None
calling timeoutHandler(): processLog=  /COMM-CENTRAL/TMP-DIR/tmpbr_pejfwpidlog
  File "/usr/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/third_party/python/sentry_sdk/sentry_sdk/integrations/threading.py", line 67, in run
    return old_run_func(self, *a, **kw)
  File "/usr/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1231, in _read
    self.timeout_callback()
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1072, in __call__
    e(*args, **kwargs)
  File "/NEW-SSD/moz-obj-dir/objdir-tb3/_tests/testing/mochitest/runtests.py", line 2917, in timeoutHandler
    traceback.print_stack(file=sys.stderr)
MochitestDesktop.handleTimeout():timeout=  3700.0
21:44.10 INFO Buffered messages finished
21:44.10 INFO TEST-UNEXPECTED-TIMEOUT | automation.py | application timed out after 3700 seconds with no output

Maybe I should re-check if my original set of patches to extend timeout values in runtests.py is still correct.
The above test did not seem to time out due to 3700 seconds timeout to be exact. I think it hit 10minutes timeout somewhere.
But I could be wrong.

Comment 201

[gene smith]

(In reply to ISHIKAWA, Chiaki from comment #200)

Hi Chiaki,
You should win the prize for 200th comment on the bug! :)
Anyhow, not sure what the connection of what you describe is to this bug (imap copy between servers of many messages at a time fails with no feedback and, on restart, copies duplicate.). Something fundamentally wrong in tb or the compiler used to build tb?
-gene

Comment 202

[gene smith]

(In reply to jwtuttle from comment #199)

Just wanted to note for all concerned that this behavior still exists in the latest version of Thunderbird, 102.7.1.

Yes, I wouldn't expect 102 to make any improvements on this. I haven't looked at this for over a year.
Are you saying you see this problem when copying between tb and a dovecot server on a local network?
Is the source folder using offline store? If so, tb won't have to fetch from the source server each message that is transferred to the (local?) destinations.
About how many messages and approximate total size are you trying to copy at once?

Comment 203

[jwtuttle]

(In reply to gene smith from comment #202)

Are you saying you see this problem when copying between tb and a dovecot server on a local network?

Most of my tests have been from one IMAP server to another IMAP server (Gmail to Dovecot, or Dovecot to Gmail, or Dovecot to Dovecot), but in several of those tests the messages were synced to Thunderbird's local store, so they did not require retrieval of each message before sending them to the destination server.

Is the source folder using offline store? If so, tb won't have to fetch from the source server each message that is transferred to the (local?) destinations.

I've tried it both using messages that were synced offline, and ones that were not. It didn't seem to make a difference. The issue seems to happen with the uploading process to the destination server, not the retrieval, I think.

About how many messages and approximate total size are you trying to copy at once?

The folder that I have been using for test purposes contains around 10,000 messages. However, I've also tried copying subsets of the messages in the folder and the silent failure issue seems to happen at a few hundred, although I haven't found exactly at what point. A 500-message selection set usually seems to fail.

Comment 204

[ISHIKAWA, Chiaki]

(In reply to gene smith from comment #201)

(In reply to ISHIKAWA, Chiaki from comment #200)

Hi Chiaki,
You should win the prize for 200th comment on the bug! :)
Anyhow, not sure what the connection of what you describe is to this bug (imap copy between servers of many messages at a time fails with no feedback and, on restart, copies duplicate.). Something fundamentally wrong in tb or the compiler used to build tb?
-gene

I was complaining about reporting how difficult to run TB under valgrind. But I might have filed the comment to a wrong bugzilla entry, I need to look at my e-mail folder to see where the message should have gone. Hmm...

In any case, I have not been able to run TB under valgrind during mochitest for a couple of years. In the past, I could in the days of |make mozmill| test framework.
I have a feeling that a few memory-related bugs have crept in and wanted to see if there are a few, or TB is free from such bugs during mochitest. This test may have suffered from such problems.

But somebody has to run TB under valgrind during mochitest, but so far I have failed due to misplaced timeout that is NOT extended properly by |mach| even if I tell it I am using valgrind (--valgrind valgrind-binary-file-name), and there can be test programs in mochitest that do not cope well with slowdown with mochitest.
Now I realize, I might have tried to post the comment to a generic entry for running TB or FF under valgrind. Sorry for the noise. I will investigate where the comment should go.

Comment 205

[gene smith]

I've made a few changes in TB that may address a main issue here. I'll fork (clone) a new bug to present my proposals there rather than extending the discussion here way beyond the 200 comment mark.
See bug 1828372.

Comment 206

[zarrafakt]

Attachment: Thunderbird error.jpg

Was experiencing this bug but disheartened to find this 14 year old post. I can successfully move up to 500 emails from local folders to another IMAP account but anymore will usually fail. I do get an error message which displays briefly and is attached for info.