1353221 - A message is marked as \deleted in INBOX is immediately and automatically expunged because server (Exchange/outlook.com or gmail) don't strictly follow imap standards, if user enabled "mark as deleted" model

Status: REOPENED

(MailNews Core :: Networking: IMAP, defect)

Description

[gene smith]

User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0
Build ID: 20161213225041

Steps to reproduce:

Use outlook.com and/or gmail imap servers


Actual results:

They don't quite act like conventional imap servers.


Expected results:

This "bug" describes what actually happens for reference.
This describes what gmail imap does but I will add a few more observations.
https://bugzilla.mozilla.org/show_bug.cgi?id=450376

Comment 1

[gene smith]

In testing Bug 1352859 I observed some seeming non-standard behavior with exchange imap server used by outlook.com and gmail imap. But that was not really the subject of that bug, so rather than clutter Bug 1352859 with this info, I am starting this report. Bug 450376 from several years ago already describes some gmail issues but I will add a few more from my point of view.

I setup a new test account at outlook.com. What I observe is that when a message is marked as \deleted in INBOX it is immediately and automatically expunged by the server. This can be seen in this manual imap transaction using openssl:

$ openssl s_client -connect imap-mail.outlook.com:993 -crlf
* OK Outlook.com IMAP4rev1 server version 17.4.0.0 ready (BAY451-IMAP77)
a login *****@outlook.com *************
a OK AUTHENTICATE completed.
a select inbox
* 3 EXISTS
* 0 RECENT
* FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
* OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags
* OK [UIDVALIDITY 14] UIDVALIDITY value
* OK [UIDNEXT 58] The next unique identifier value
a OK [READ-WRITE] SELECT completed.
a uid fetch 1:* (FLAGS)
* 1 FETCH (FLAGS (\Seen) UID 44)
* 2 FETCH (FLAGS (\Seen) UID 45)
* 3 FETCH (FLAGS (\Seen) UID 48)
a OK FETCH completed.
a uid store 44 +flags (\deleted)   <------ "delete" the email at UID 44 (sequence 1)
a OK STORE completed.
a noop
* 1 EXPUNGE                        <------ expunge reported, non-standard behavior!
* 2 EXISTS
a OK NOOP completed.
a uid fetch 1:* (FLAGS)
* 1 FETCH (FLAGS (\Seen) UID 45)
* 2 FETCH (FLAGS (\Seen) UID 48)  <------- now only 2 mails listed in INBOX
a OK FETCH completed.

This transaction marks as deleted the message at UID 44. The noop indicates it has also been expunged and the exists count is now 2; and the final fetch verifies there are now only 2 emails in the INBOX instead of the original 3. This is essentially what TB does when a message is "deleted" in TB with the "just mark as deleted" imap model.
When a delete operation on outlook.com INBOX occurs in TB, again with the "just mark as deleted" mode set, the email becomes crossed out with a red X in front as expected. However, after a few seconds the X'd out marking disappears and the message appears to still be in the inbox. If another folder is selected and then inbox is re-selected or "get messages" is clicked, the "deleted" message goes away and it has not been moved to another folder of the outlook.com account; it was permanently deleted (expunged).
Note: Although outlook.com does not have the MOVE capability, the newly added UidExpunge() from bug 1231592 is not occurring in this scenario since a move is not happening.

In gmail with the "just mark as deleted" model enabled, deleting emails in folders other than [Gmail]/All Mail also causes the server to expunge the marked emails automatically like outlook.com. However, the emails are still present in "All mail", which is OK. If email in "All mail" is deleted it is just marked as deleted and not automatically expunged. Even an expunge (compact) on "All mail" folder does not delete the emails marked as deleted (they remain visible and X'd out). To actually delete email completely from the gmail account you must select and drag (move) the emails from any source folder to [Gmail]/Trash. Then in Trash, delete the emails (X'd out) and compact Trash. This seems to truly delete (expunge) emails from the gmail account in all folders, including [Gmail]/All Mail.
Note: When emails are dragged to Trash, they must NOT be already marked as deleted (X'd out). The deletion and compaction must occur in [Gmail]/Trash. 

In conclusion, this is not really a "bug". However, it might seen as useful information. If anything, it show why imap servers that don't strictly follow standards may cause TB to behave in unexpected ways.

Comment 2

[Alfred Peters [:infofrommozilla]]

(In reply to gene smith from comment #1)
> In testing Bug 1352859 I observed some seeming non-standard behavior with
> exchange imap server used by outlook.com and gmail imap. [...]

> I setup a new test account at outlook.com. What I observe is that when a
> message is marked as \deleted in INBOX it is immediately and automatically
> expunged by the server. This can be seen in this manual imap transaction
> using openssl:

Gmail has a configuration setting for this (translation from German):

Settings -> /Forwarding and POP/IMAP\

=> If I mark a message in IMAP as deleted:
   (x) Auto-delete - Update server immediately (default)
   ( ) Auto-delete - Wait for the client to update the server

Comment 3

[gene smith]

(In reply to Alfred Peters from comment #2)
> 
> Gmail has a configuration setting for this (translation from German):
> 
> Settings -> /Forwarding and POP/IMAP\
> 
> => If I mark a message in IMAP as deleted:
>    (x) Auto-delete - Update server immediately (default)
>    ( ) Auto-delete - Wait for the client to update the server

Just now checked into this for gmail. Yes, you are right. When I set it to "...Wait for client..." I can "delete" an email and it stay in place with the red X and a line through it and doesn't get instantly expunged. Thanks for pointing this out!

Don't see a similar setting for outlook email, that uses excahnge, however.

Comment 4

[gene smith]

Wayne, I created this "bug" on Jorg's suggestion as mostly documentation. It really only affects users that use the non-default "just mark it as deleted" method for deleting messages (aka, "imap delete model"). I'm don't think anything can really be done about it. I have discussed this with a gmail developer on the imap mailing list and he pretty much convinced me that the behavior pointed out here is not actually a violation of the imap RFC. So, I think you can close this with reason WONTFIX if that's possible. 
Thanks,
-gene

Comment 5

[Wayne Mery (:wsmwk)]

gene, thanks for researching this!

Comment 6

[gene smith]

I had my Outlook account set to "Just mark as deleted" and was trying to clean up some old test emails I had there. Each time I selected a bunch of emails and deleted them the strike-through appeared and then quickly went away. I thought it was a bug in TB causing this but saw that the messages were getting auto-expunged as soon as they were marked \deleted. Then I remembered this bug.

A while back I also received an email from a user experimenting with this feature on an o365 account and ended up loosing a bunch of emails since they weren't aware that the auto expunge would occur. So this can be a problem for users that might just want to see what "mark as deleted" does.

Maybe this could be handled in Account Manager so when an outlook.com or gmail.com server is configured, this option is disabled. However, for only gmail, you can disabled it at the webmail site so the message isn't expunged when you mark it with \deleted flag.

So, to keep flexibility, I'm suggesting just a change in the wording of the server setting "Just mark it as deleted" to read something like:
Just mark it as deleted. Warning: Some servers do an immediate removal.

This should provide sufficient caution to users that this non-default option might cause problems without mentioning specific servers or the somewhat technical term "expunge". Also, it needs to be short so it doesn't wrap.

Comment 7

[gene smith]

Attachment: Bug 1353221 - When using imap "mark as deleted" warn that messages may still be expunged. r=mkmelin

Comment 8

[betterbird.project+13]

It looks a little strange to have one sentence end with a full stop and then second one without. Can you make it one of these:
Just mark it as deleted. Warning: Some servers do an immediate removal.
Just mark it as deleted - Warning: Some servers do an immediate removal
Just mark it as deleted. Warning: Some servers do an immediate removal!

Comment 9

[betterbird.project+13]

From Phab:

It's interesting outlook.com addresses have this behavior, but I don't see it for office 365, even if both use outlook.office365.com.

Paid O365 tenants show this behaviour. The marked message is purged after a split-second :-(

Comment 10

[Magnus Melin [:mkmelin] (away until Aug 10)]

Not for the account I have, and I actually do use mark-as-deleted for. Perhaps there is some organizational setting.

Comment 11

[gene smith]

I'm not seeing auto-expunge on the o365 test account either (provided a while back by Magnus). Seeing it on my free outlook account and, of course, on gmail.