1828372 - When moving multiple messages to an IMAP server, mark as deleted each individual successfully copied messages

Status: RESOLVED FIXED

(Thunderbird :: Folder and Message Lists, defect)

Description

[gene smith]

Attachment: bulk-imap-move-v0.diff

+++ This bug was initially created as a clone of Bug #538375: "When attempting to copy or move large numbers of local messages to an IMAP server, it fails without any error or status message. Related to timeouts" +++

Currently when multiple messages are "moved" to an imap folder from a different account (i.e., another imap account, pop3 account or from a Local Folder) each message is sequentially read from the the source folder into a temp file and the file is sent to the destination folder on the server using imap APPEND command triggered by "appendmsgfromfile" imap URL. Once all the message have been copied, they are, only then, marked deleted to complete the "move".

My proposed change is to mark each "moved" message as deleted immediately after the individual message has been successfully copied to the destination folder. This way, after a network or imap server error causes the move to stop, you can tell which messages were copied successfully in the source folder since they are either gone or, if delete method "just mark as deleted" is set, they are crossed out. Then the move can be restarted by selecting the not yet moved messages and again move the remaining messages to the desired destination.

Of course, doing the multi-message move by copying all the messages and then deleting all the source messages in one command is faster, which is probably why the code currently does that since there will be much fewer round-trips to the server. So it may be possible to preserve this optimization as the default behavior and enable my proposed change via a pref. But I doubt that the speed is that important if a user moves only a few messages, which is typical. Note also that multiple message copies won't be affected by my proposed change since the source messages are not deleted.

Testing with yahoo as the destination imap server, it appears that about 5000 messages can be moved to the server before it complains with a * NO [LIMIT] Rate limit hit error which terminates the transfer. This is a typical response by yahoo when it thinks TB is abusing it. I've seen this error also sometimes when TB tries to make more than 3 imap connections to yahoo server. It really has nothing to do with commands coming in at a too fast rate.

It's also possible to drag a folder from account A and drop it to a folder on account B (or to account B itself). Currently when this is done between accounts/servers, the transfer is forced to be just a copy (the source folder(s) and messages remain untouched) so moves are not possible which doesn't help with this bug. Also, drag/drop of folder(s) is the only way to do a recursive folder copy so that a folder's sub-folders and their message are also brought along by the copy.

To fix this so that recursive move of all the messages in the folder and its sub-folders is possible I made a change so that if a move during the drag is detected (e.g., user has not held down ctrl key) still after each message is copied successfully it is marked deleted on the source folder but the source folder and its sub-folders remain present. So on a successful "move" using drag/drop, the source folders are empty of messages (or contain all marked as deleted / crossed-out messages). Effectively the change copies the folder structure to the destination while moving only the messages.

Note: the reason that drags between accounts only do a copy and never a move has something to do the possible effect on filters, but don't know the details. My proposed change doesn't affect the source folder structure (only deletes the messages) so it shouldn't cause a problem for filters. (But there may be unit test that proves me wrong.)

There is another case where an imap folder is dragged to a Local Folder instead of to another imap account/server. With the current code, again this does a pure copy of folders and messages into the destination Local Folder. So it's not possible to tell which messages may have failed to copy since they are never deleted from the imap source folder. With my proposed change, each message in the source imap folder is deleted only after a successful file copy to Local Folder occurs. This most typically will result in messages remaining present in the source imap folder when a network error prevents imap fetching of the messages to be written to the destination Local Folder. (Note: this is probably less of an issue when source imap folder uses offline store since, in that case, the transfer is just file to file with no network access required.)

Attached is my preliminary changes required.

Comment 1

[gene smith]

Attachment: bulk-imap-move-v3.diff

After a lot of testing and debugging and code tweaks, here an almost finished diff. It still contains all my printf's and commented-out alternatives. As is, it works as I intend and passes the relevant mochi and unit tests. I'll submit a cleaned up formal moz-phab diff, with the same functionality, shortly.
Note: the changes to the test files are due to a new parameter added to copyFolder() to allow moving of messages without moving the source folder.

Comment 2

[gene smith]

Attachment: Bug 1828372 - When moving messages across servers, mark each message as deleted on success. r=mkmelin

6-5-2023: Patch approved for landing.

Comment 3

[gene smith]

This has now been moved into attachment move.pdf

How to move messages between IMAP server and others (with comment 2 patch and bug 1831759 changes in effect):

If the source folder is IMAP, under Settings > Server Settings temporarily set the delete method to "Just mark as deleted", disable "Check for new mail every X minutes" and disable "Check for new mail at startup". Do the same for the destination account (except setting "Just mark as deleted" is not necessary). With these setting, as each source folder message is copied to the destination server, it will be crossed out but each message will remain visible in the list of messages. Each message is set as IMAP "deleted" at the server but is still actually present on the server and in TB.

After these configuration changes are made, it is recommended to restart TB to ensure they take effect and to reset all connections to the IMAP servers.

Gmail fix for "Just mark it as deleted"

Some IMAP servers like gmail completely delete (expunge) messages marked as deleted from the server by default. But there is a setting at gmail site for each account under Forwarding and POP/IMAP and then under IMAP access called When I mark a message as IMAP deleted. Set this to Auto-expunge OFF. Be sure to click the Save Changes button at the bottom of the gmail setup page for the change to take effect in the gmail server. Once moving of messages from gmail to another server is complete, setting Auto-expunge ON again should be done if that was the original setting.

But even if the messages marked deleted are expunged by the server, the message won't be deleted until it is copied to the destination server. So this is just an extra backup for data loss security.

Disable Message Threading

On the source account to be moved to the destination, select the column icon Toggle Message Threads (typically located at the far left end of the message list column heading) so that message threading is disabled and the messages all appear as a flat list. This is to ensure that the message order numbers (enabled in the next step) are always in ascending order.

Setting "Order Received" for source folder

On the source account folder to be moved to the destination, select the column icon Select columns to display (typically located at the far right end of the message list column headings) and select Order Received from the list and the new Order Received column will appear. Sort the messages by clicking on the Order Received column. Copying messages always occurs from lowest to highest (ascending) "Order Received" value. Note: the values may not start at 1 and there may be gaps since, for IMAP, "Order Received" is the message UID number set by the server.

There are four ways to move messages from a source to a destination server:

1. Select all or some messages in a single folder and move them to an existing folder in the destination account using the right-click message context menu item Move To > and then select the destination folder.
2. Select all or some messages in a single folder and move them to an existing folder in the destination account using drag and drop to the destination folder.
3. Select a single folder from the folder list and using the right-click folder context menu item Move To > to select the destination folder or account. This will also move all sub-folders of the selected source folder.
4. Select a single folder from the folder list then drag and drop the source folder (and its sub-folders) to the destination folder or account.

When moving messages with method 1 or 2, a new folder is never created at the destination. You have to create the folder ahead of time or just move the messages into an already existing folder.

Method 3 and 4 copies the source folder structure to the destination folder and creates a new folder (and possible sub-folders) with the same name. Method 3 and 4 doesn't add messages into an existing folder. The destination for the folder move can be another folder in the destination account or it can be the account itself which will create a new top-level folder.
With method 3 and 4, the source folders remain in place and only the messages are "moved" (actually, just marked deleted and will display as crossed out in the source folder message list).

With all the methods, the source folders can be watched in TB to see the progress. With messages sorted before the move by "Order Received", as each copy to destination occurs each individual message in ascending order will be crossed out (if recommended "just mark as deleted" is configured; otherwise, the message will just disappear, also in ascending order).

Error recovery

So if the network connection to the server goes away or if the server reports an error that causes the transfer to stop, notice the first message not yet marked out (or still visible if not using "just mark as deleted"). Remember the "Order Received" value for that message or just flag it in the source message with a "Star" or set a tag (like "Important") on that message as a placeholder. This is the first messages that didn't get marked as transferred.

Next, restart TB and select the destination folder and disable message threading and set the Order Received column like was done for the source folder and sort in ascending Order Received. Since TB uses a "streaming" model to do the transfers when the source messages are in offline store (the default mbox or in maildir), it's likely that a few more messages got transferred with a higher "Order Received" numbers than the first one not marked deleted in the source folder (the one just "starred" or tagged). So if more messages got transferred, find and remember the subject and date/time of the highest "Order Received" message in the destination folder. Now go back to the source folder and find the actual last message transferred that has that matching subject and time/date This is the actual last successfully transferred source message. Select that messages and all the others between it and the message with the star or tag (including the starred/tagged message) and "delete" these messages. This should just sets the "deleted" flag on these and crosses them out and doesn't remove or delete them from the server or from TB.

Now select the remaining not crossed-out messages in the source folder and using method 1 or 2 above, move the selected messages to the destination folder. If errors occur again, just repeat the recovery steps as necessary to complete the transfer of the full source folder to the destination folder.

By following this procedure it should mostly avoid copying duplicate messages into the destination folder and also avoids having to completely start over after an error halts the transfer.

Notes

Note1: if "just mark as deleted" is not selected and "move to trash" (the default) remains selected, after a successful message move the message is still marked as deleted and not moved to trash and is still present but is not visible. But, if "just mark as deleted" is then selected, the message should still be present and visible but crossed out. Of course, this depends on the server not "auto-expunging" messages marked as deleted, e.g., gmail with non-default setup described above.
/
Note2: Some imap servers (for example Charter/Spectrum "Intermail") don't keep the "deleted" flag set on the messages after a re-connect, as occurs on TB restart. So the first not yet moved and lowest numbered "Order Received" must be noted (or starred or tagged) before restarting TB since messages previously shown as marked out will, after restart, appears again as normal (not marked out) messages.

Move from IMAP folder to Local folder:

Moving messages from an IMAP source folder to Local Folders or a POP3 folder is, of course, also possible. If the IMAP source folder has full offline store (mbox or maildir), the move could actually be done with TB offline since network access is not required. But if the IMAP source folder does not have offline store, an IMAP fetch of all the messages occurs in one (potentially giant) IMAP fetch command and the full messages are streamed to the local filesystem used by Local or POP3 folders. As each message is successfully written to the destination file, it is also marked deleted at the IMAP source folder so if the network goes down before the transfer of all messages finishes, the messages not yet marked deleted (not yet crossed out) can be moved again (after a recommended TB restart).
However, there is one issue with IMAP to Local/POP3 moves in that the IMAP server does not immediately receive the IMAP signal to mark the message deleted, only the TB database receives it. So before restarting TB, notice the first "Order Received" message number that was not marked deleted (not crossed out). Then on TB restart, move only the messages with that number and higher to avoid moving duplicates since the already moved messages will appear again not crossed out and fully visible. Also, as described in the "Error Recovery" section above, it's possible for a few more message to get copied, so check in the Local Folder destination for the possible additional messages that shouldn't be re-copied.

Move from Local folder to IMAP:

Moving message from Local Folders or POP3 folder to IMAP can also occur. This is similar to IMAP to IMAP cross server transfer except the source Local/POP3 folder can't be marked as deleted and each message is literally deleted/removed from the Local/POP3 folder after a successful copy to the destination IMAP folder. So sorting these by "Order Received" before doing the move may not be as useful. If the network fails during the transfer, just restart TB and move again the remaining not yet deleted messages in the source local folder.
Since the messages are permanently deleted from the local folder when moved, be sure to have a backup of the Local/POP3 folder before doing the move to IMAP.

(Addendum) Folder drag/drop and context copy/move policies:

• Can't drag or copy/move a "special" folder, i.e., one having a special identifying icon such as Inbox, Sent, Junk, Trash etc.
• Within an account a non-special folder can only be moved; folder copy (or drag with ctrl pressed) silently becomes a move.
• Drag (with ctrl pressed) or copy of a non-special folder between accounts is allowed.
• Drag (without ctrl pressed) and move of non-special folders between accounts is allowed but only the messages are moved.
• Drag to move or copy a folder also brings along all the sub-folders. (TBD: allow for non-recursive move/copy?)
• Observation: drag/drop to move or copy a folder, to many users I suspect, is an unknown feature.

(Addendum) Note regarding outlook.com:

If a folder and its possible sub-folders are copied or moved to an outlook account, URL "ensureexists" succeeds (the new folder is created and subscribed to with imap OK tagged responses). However, next when the imap append occurs to copy in the first message to the new folder, outlook responses with "NO [TRYCREATE] even though the new folder is present. This seems to be timing related since if the append is delayed some the initial append and subsequent appends into the new folder succeed.

Comment 4

[gene smith]

Re: bug 538375 comment 206
zarrafakt,
When you say "move from local folder", do you mean literally TB's "Local Folders" or maybe just another locally stored imap folders?
The error you see from outlook.com I suppose just means the server is refusing to accept any more messages, so my proposed changes here should allow you to restart the move with only the not yet moved messages selected.
I'll try moving more than 500 message to outlook and see what happens for me...

Well, at least for me here, I've been able to move a folder with more than 1000 messages to outlook account with no errors reported by the outlook server. One of the messages was about 25M in size, the others were more typical size (bugmails from mozilla).

Just to be clear, the changes proposed in this bug report are not yet in any TB version -- daily, beta or formal release.

Comment 5

[zarrafakt]

(In reply to gene smith from comment #4)

Re: bug 538375 comment 206
zarrafakt,
When you say "move from local folder", do you mean literally TB's "Local Folders" or maybe just another locally stored imap folders?
The error you see from outlook.com I suppose just means the server is refusing to accept any more messages, so my proposed changes here should allow you to restart the move with only the not yet moved messages selected.
I'll try moving more than 500 message to outlook and see what happens for me...

Well, at least for me here, I've been able to move a folder with more than 1000 messages to outlook account with no errors reported by the outlook server. One of the messages was about 25M in size, the others were more typical size (bugmails from mozilla).

Just to be clear, the changes proposed in this bug report are not yet in any TB version -- daily, beta or formal release.

Hi Gene,

I am moving email from a POP3 account (all emails on PC) to an IMAP account. 25 years worth but all in all under 100,000 emails (many with attachments and some >30 MB). Sub 500 had been my workable limit until this morning when it started failing at lower numbers. So I moved one at at time until I found one email that refused to move. Status line at the bottom of the TB window reported "Checking Mail Server Capabilities" and then just gave up. I deleted that email and now I'm back to being able to move up to 500 at a time (it's a slow process, too scared to go higher, I started on Friday and am up to 2022). The problematic email had 6 attachments (.pdf, .dwg & .dxf files) with a total filesize of 4.0 MB.

Comment 6

[gene smith]

Ok, thanks for the info.
If you would like, I can make a "try" build with my proposed changes that you can use to move your POP3 message to outlook.com IMAP. You should be able to select them all and just let it do the move and if it fails or stops for some reason, just restart TB and select the as yet not moved messages and start the transfer again.
A try build would be based on the current daily TB version so I need to know your OS type (windows, linux etc).
I've written a "guide" for how to do the moves in comment 3 above. Your specific transfer type is described in the "Move from Local folder to IMAP" section. I just added to the section that you should probably have a backup before doing the move since the messages actually get deleted from the local folder after they are each individually transferred to the IMAP server.

Comment 7

[zarrafakt]

Many thanks for the offer but I fiinished transferring all files late last night and I won't be doing that again in a hurry. Of whatever use this may be I noticed the following during the process,

1. The next time I had a fail I counted back through the messages that did copy over and found the "one" that was the problem. Tried to move it on it's own but no joy. The message was around 4MB and had again .dwg & .dxf files as attachments. I deleted the two .dxf attachments and it would then move. I googled .dxf and Outlook and there seems to be a history of issues.

2. Several other fails happened as I worked through always with attachments although they were other filetypes (.txt, .pdf). When I tried to move each message on its own it would fail but then I discovered I could copy them over.

3. Another error message would pop up this time referring to "Command argument error 11".

Anyway, I'm out, best of luck with your endeavours.

Comment 8

[gene smith]

Ok, no problem and glad you were able to finally complete the transfer.
Bulk transfers to another server isn't something user do every day so that's probably why the issues have fallen through the cracks over the years regarding this.
Anyhow, I attempted to make a "try" build with my changes yesterday in case you wanted it but the try-comm-central server was in in a "busted" state for unknown (to me) reasons.

Comment 9

[gene smith]

Attachment: move.pdf

Comment 10

[gene smith]

Attachment: move-src-for-pdf.odt (set obsolete, used to update move.pdf)

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/f6278cd8e103
When moving messages across servers, mark each message as deleted on success. r=mkmelin,benc

Comment 12

[Geoff Lankow (:darktrojan)]

Backed out for test failures:
https://hg.mozilla.org/comm-central/rev/f33044e69a0bedf4889409f9a167c3cf8d6cc1da

Comment 13

[gene smith]

(In reply to Geoff Lankow (:darktrojan) from comment #12)

Backed out for test failures:
https://hg.mozilla.org/comm-central/rev/f33044e69a0bedf4889409f9a167c3cf8d6cc1da

I've updated the patch to fix these test errors and requested review. They all now pass the subject tests but two of the tests in the "extension" area still have a problem while the test is shutting down. I don't think my changes are affecting this since there are existing bug reports for these tests that mention this.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/7fa895a86b55
When moving messages across servers, mark each message as deleted on success. r=mkmelin,benc

Comment 15

[gene smith]

Attachment: move.pdf

Added description for when setting mail.imap.expunge_option = 3 is needed to never "auto expunge" when number of marked deleted messages reaches the default threshold of 20.

Comment 16

[gene smith]

Attachment: move-src-for-pdf.odt--v2

Comment 17

[Wayne Mery (:wsmwk)]

Comment on attachment 9330453 [details]
Bug 1828372 - When moving messages across servers, mark each message as deleted on success. r=mkmelin

[Triage Comment]
Approved for beta

Been on nigthly almost week

Comment 18

[Daniel Darnell [:dandarnell]]

Thunderbird 115.0b4:
https://hg.mozilla.org/releases/comm-beta/rev/28a3731793d9

Comment 20

[gene smith]

Attachment: move-v2.pdf