1383033 - Incorrect synchronization of moved items to IMAP server

Status: RESOLVED INCOMPLETE

(Thunderbird :: Folder and Message Lists, defect)

Description

[Dami]

User Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36

Steps to reproduce:

Create a subfolder of Inbox.
Move an email item to the subfolder.



Actual results:

The email item has been correctly moved to the subfolder in the Thunderbird interface. However, on the server side, copy of the email item has been created in the subfolder and the item has also stayed in the inbox. Only upon exiting Thunderbird, the "Inbox" copy of the item has been deleted from the server.



Expected results:

The move operation should have been fully completed at once, not only after existing Thunderbird.
Tested on FRESH INSTALL of Thunderbird 52.2.1 32bit, Windows 8.1, IMAP server outlook.office365.com
Similar behavior seen on iMac with the same IMAP server.

This behavior was working as expected in version 45, and is working as expected in 55.0b2. It even work as expected, when updated from version 45 to 52.2.1. Issue has been observed only on fresh install.

Comment 1

[Jorg K (CEST = GMT+2)]

More for Gene to look at ;-)

Comment 2

[gene smith]

I see the same thing with 52.2.1 as currently installed with charter email. This is because when email "moved" (actually, copied) from INBOX to INBOX/SubFolder the message remains in INBOX but marked /deleted but not expunged. It still appears to be present in INBOX when looking at charter webmail since charter webmail apparently doesn't respect the /deleted flag. A fix for this is in the beta version using a call to UidExpunge() since charter supports UIDPLUS but does not support MOVE.

I don't have an outlook.office365.com account since it doesn't seem to be free (*). The closest I have is an outlook.com account and it seems to work with no problems, unlike charter with 52.2.1. I can move a message from INBOX to INBOX/SubFolder and on outlook webmail the message disappears from INBOX as expected and appears in INBOX/SubFolder. I haven't checked the imap log but I suspect outlook.com supports MOVE. I did not do a fresh install but I did *completely* delete the outlook.com account, including all the data, and reconfigured it. No problem seen.

> Only upon exiting Thunderbird, the "Inbox" copy of the item has been deleted from the server.

This was also observed with charter email. Exiting tb causes an imap CLOSE to be sent to the mbox which expunges the email. So, outlook.office365.com seems to act a lot like charter. However, I don't understand why problems only occur with fresh install and not with upgrade. 

If possible and if not already tried, could the reporter do a "fresh install" of 55.0b2 since this fixes problems seen with charter imap that sound similar to the outlook.office365.com issue.

Can reporter tell me if there are other accounts that work OK at a fresh install and if only issue is with outlook.office365.com? Also, is it possible that removing the account and its data and re-establishing the account also causes the problem on an existing, non-fresh installed tb?

Also, unless I can obtain a trial or guest account and test it myself (*), if the reporter could provide an imap log for outlook.office365.com it would be most helpful:  https://wiki.mozilla.org/MailNews:Logging

(*) Don't see a way without providing MS a credit card number and/or educational institution ID.

Comment 3

[Alfred Peters [:infofrommozilla]]

See also: Bug 1353221 and Bug 1352859

Comment 4

[gene smith]

Bug 1353221 documents that outlook.com automatically expunges emails deleted. This is only a problem if delete model is "only mark as deleted". Gmail does the same by default but has a configuration option at their site to not do the automatic expunge so that the email only has /deleted flag set which is standard behavior. Unfortunately, outlook.com doesn't seem to have that option at their site.

However, not sure if any of this actually relates to the reporter's issue with outlook.office365.com and the fresh install problem. Need more info from reporter.

Comment 5

[Dami]

Attachment: Issue_imap.log

Log from the session where the issue has been observed.

Comment 6

[Dami]

Attachment: Synced_imap.log

If there is already an email item with the same subject in the destination folder, the deletion from the source folder is immediate. See the logfile.

Comment 7

[gene smith]

(In reply to Dami from comment #6)
> Created attachment 8889324 [details]
> Synced_imap.log
> 
> If there is already an email item with the same subject in the destination
> folder, the deletion from the source folder is immediate. See the logfile.

Thanks for the logs. It appears that the 2nd log shows a different imap CAPABILITY response in that it shows MOVE is supported. The first log you provided shows no MOVE support. The imap ID response for both logs show the same server version so this doesn't make sense to me. Why would capabilities vary between two thunderbird start/terminate cycles for the same imap server?

In the first log (attachment 8889322 [details] line 950), the "move" to the subfolder uses imap COPY followed by imap STORE /deleted flag for the copied item. This will cause the message stored at the server to just be marked as deleted but not yet fully deleted (i.e., not expunged). So whether it appears deleted at the server (I assume you are using a webmail app) depends on whether the /deleted flag is respected by the application. Apparently it is not. At line 1034 of the first log an imap CLOSE occurs for the INBOX and INBOX/tst that, per the imap RFC, will expunge any message having the /deleted flag set in those mailboxes. Close occurs when tb is shutdown so the /deleted messages in the closed mboxes will be fully deleted and disappear in the application.

In the 2nd log (attachment 8889324 [details] line 957), a real imap MOVE to the subfolder occurs which copies the messages and then reports an automatic expunge by the server. So in this case the message appears to instantly disappear from the source mailbox due to the expunge by the server. I don't think this has anything to do with "subject" text of the email.

I did a "fresh install" of tb 52.2.1 on OSX (an old Macbook Air) and setup my outlook.com account. The capability there also reports as supporting MOVE and I see no problem when looking at outlook.com webmail while doing a move to INBOX/tmp. The email instantly disappears from INBOX when moved to INBOX/tmp by tb. I suspect that the "fresh install" theory might be a "red herring" and the real cause is the variable MOVE support.

Another experiment you can do is just delete a message from INBOX. Does it go away in the webmail app? This should only work when MOVE capability is reported and you are using the "When I delete a message, move it to this folder:" server option in tb.

Also, please record several more logs during tb start/stop cycles and see how many show MOVE capability. You don't have to post them. If MOVE support is truly random, the only immediately solution would be to use the tb beta version since it simulates "move" using COPY followed by the UIDPLUS's UID EXPUNGE imap command (provided capability UIDPLUS is supported, which it is according to your logs). Just to be clear, the capability response line in the log looks like this:

2017-07-24 08:30:53.526000 UTC - 19288[144aafb0]: 144dd800:outlook.office365.com:NA:CreateNewLineFromSocket: * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+

Thanks!

Comment 8

[gene smith]

Another possible workaround without using the beta is to set the preference "mail.imap.expunge_after_delete" to true with the config editor. This is a big hammer that will immediately expunge *all* messages marked with /deleted in a mailbox when tb deletes (actually, marks as /deleted) any given message. This defaults to false since it can cause issues when one or more clients access a mbox and at least one uses delete mode "Just mark as deleted"; all marked emails will (mysteriously) disappear at a client when another client deletes any message. But if you are using the default delete mode (Move it to this folder...), which I think you are, this is probably not a problem.

Comment 9

[Dami]

Thank you Gene, you are right. The behavior is different depending on the server capability. About one fifth of the connections to outlook.com are to a server without the MOVE capability. My "clean install" observations were mere coincidence. Deletion is behaving similarly, the deleted item is still visible in webmail.

I will file a support request to Microsoft about the inconsistency of their server capabilities and post the results here.

As for the beta version workaround using UIDPLUS - is it just a specific beta version behavior or we can expect it to be a default in future production releases?

Comment 10

[Dami]

Also, I have noticed an interesting thing - the reported MOVE capability can change during TB session. However, even if MOVE is starts to be supported, the behavior stays the same. It works as expected only if the MOVE capability was reported on first capability check.

2017-07-25 06:50:07.451000 UTC - 7740[14f393a0]: 1305d800:outlook.office365.com:NA:CreateNewLineFromSocket: * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+

2017-07-25 06:50:07.451000 UTC - 7740[14f393a0]: 1305d800:outlook.office365.com:NA:CreateNewLineFromSocket: 1 OK CAPABILITY completed.

2017-07-25 06:50:07.779000 UTC - 7764[14f39cd0]: 13065800:outlook.office365.com:NA:CreateNewLineFromSocket: * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+

2017-07-25 06:50:07.779000 UTC - 7764[14f39cd0]: 13065800:outlook.office365.com:NA:CreateNewLineFromSocket: 1 OK CAPABILITY completed.

2017-07-25 06:50:39.715000 UTC - 2916[a7bbc10]: a72f800:outlook.office365.com:NA:CreateNewLineFromSocket: * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+

2017-07-25 06:50:39.715000 UTC - 2916[a7bbc10]: a72f800:outlook.office365.com:NA:CreateNewLineFromSocket: 1 OK CAPABILITY completed.

2017-07-25 06:59:06.941000 UTC - 7868[ab0a4f0]: d47d800:outlook.office365.com:NA:CreateNewLineFromSocket: * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+

2017-07-25 06:59:06.942000 UTC - 7868[ab0a4f0]: d47d800:outlook.office365.com:NA:CreateNewLineFromSocket: 1 OK CAPABILITY completed.

2017-07-25 07:00:55.193000 UTC - 7340[517ac0]: 9d2f000:outlook.office365.com:NA:CreateNewLineFromSocket: * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+

2017-07-25 07:00:55.193000 UTC - 7340[517ac0]: 9d2f000:outlook.office365.com:NA:CreateNewLineFromSocket: 1 OK CAPABILITY completed.

2017-07-25 07:04:10.363000 UTC - 5028[a7b7680]: d476000:outlook.office365.com:NA:CreateNewLineFromSocket: * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+

2017-07-25 07:04:10.363000 UTC - 5028[a7b7680]: d476000:outlook.office365.com:NA:CreateNewLineFromSocket: 1 OK CAPABILITY completed.

Comment 11

[gene smith]

Good to confirm that. It does seem strange that the MOVE feature is variable! Curious what their support says. 

Yes, the feature to simulate MOVE with COPY-DELETE-UidExpunge will be in the official release, not just the beta. The information I have says it will appear in version 55.0 but I don't know when that will be released.

Tb probably does a CAPABILITY query for every mailbox and some may see the MOVE and others don't. Tb maintains tcp/ip connections to 5 mailboxes by default and each has a capability query done on it. Also, when a 6th mbox is accessed another CAPABILITY query occurs since a connection has to be closed and a new one opened. I don't think a given capability response is used globally (for all mboxes) so seeing a MOVE later in a response as you show won't necessary apply to INBOX and INBOX/tmp that you have been focusing on. Then again, I could be wrong since I haven't studied this in detail.

Comment 12

[gene smith]

With outlook.com I also see variable MOVE support, but some mailboxes report MOVE (e.g., INBOX, INBOX/sf) while others don't (INBOX/tst) but it seems consistent per mbox. However, with outlook.com they have a non-standard IMAP feature that automatically does a expunge on any message after it's marked as /deleted. So deleted messages disappear in webmail even if the mbox doesn't support MOVE. I can't find on the outlook.com webmail page a way to disable this non-standard behavior.

Since you have the more "professional" outlook service, you might check if it has a configuration feature to turn on/off expunging (maybe called permanent delete) after email is deleted. (Gmail also has the expunge on delete feature enabled by default but you *can* turn it off at your gmail webmail page.)

Comment 13

[Dami]

Hello,
Microsoft support has confirmed that there is no way to configure expunging. As for the consistency of MOVE support they have not been very helpful - Outlook as IMAP client behaves consistently so they do not see it as a problem (and other clients and IMAP standards are obviously not so interesting).

Thank you for your help, we will be using the mail.imap.expunge_after_delete workaround until the version 55 is available.

Comment 14

[Jorg K (CEST = GMT+2)]

There seems to be some confusion here. The new feature from bug 1231592 and bug 1360117 is available in TB 55 beta and TB 56 beta, and I'm still not sure whether I will include it in a TB 52.x release since it changes the delete behaviour, see bug 1352859. I'd rather not have complaints for a dot release.

Comment 15

[Wayne Mery (:wsmwk)]

Dami,do you stlil see this issue when using version 60?

Comment 16

[Wayne Mery (:wsmwk)]

If incomplete isn't the correct status for this, feel free to make a correction