Messages unexpectedly expunged after move

NEW
Unassigned

Status

MailNews Core
Networking: IMAP
--
critical
5 months ago
8 days ago

People

(Reporter: Magnus Melin, Unassigned)

Tracking

({regression})

regression

Thunderbird Tracking Flags

(thunderbird55 affected)

Details

(Whiteboard: [regression:TB55])

(Reporter)

Description

5 months ago
For some reason I hadn't noticed this before. Should be from bug 1231592.

Now when I move a message to another folder it's immediately removed and expunged. I'm using the mark-as-deleted imap model, so that's why it's showing, and it's unexpected.

And mail.server.default.forceSelect is false by default, so the behaviour should not have changed.


Expected result: should not immediately expunge a message after move.

Comment 1

5 months ago
Hmm, I'm surprised. We took great care only to expunge if the server doesn't support MOVE:
https://hg.mozilla.org/comm-central/rev/4fd458dcdd4a95a833e577ed0ef2e2b0d176b85e#l7.64
Says: !kHasMoveCapability

https://hg.mozilla.org/comm-central/rev/4fd458dcdd4a95a833e577ed0ef2e2b0d176b85e#l7.85

Which server are you using, Magnus? Gmail? that should support move that there shouldn't be any difference. What am I missing?

Gene?
Flags: needinfo?(gds)
Version: 38 → 55
(Reporter)

Comment 2

5 months ago
It's an Exchange server actually.

From https://hg.mozilla.org/comm-central/rev/4fd458dcdd4a95a833e577ed0ef2e2b0d176b85e it looks like you call UidExpunge() when gExpungeAfterDelete is false. That's probably why. Earlier we did nothing.

Comment 3

5 months ago
Sure, we call UidExpunge(messageIdString), so just expunging the message that got moved on a server that didn't support MOVE, so had to be copied and might as well be expunged from the original source.

Does that Exchange server really not have MOVE capability? You saw that all this is controlled by:

             if (GetServerStateParser().LastCommandSuccessful() &&
               (m_imapAction == nsIImapUrl::nsImapOnlineMove) &&
               !(GetServerStateParser().ServerIsAOLServer() ||
                 GetServerStateParser().GetCapabilityFlag() & kHasMoveCapability))

So if the server offers MOVE, that UidExpunge(messageIdString) won't run.

Sure, for non-MOVE servers the behaviour has changed, but is it really not better now? You *move* something, so why keep a superseded copy in the old located? Try on Gmail, which I understand supports MOVE, and you won't see that.

Comment 4

5 months ago
gmail's even worse. If you mark a message as deleted then move to another folder and then come back, the message goes away. On charter, I can move marked as deleted messages OK but haven't yet tried what I did on gmail. I will keep looking...
Flags: needinfo?(gds)

Comment 5

5 months ago
Seems OK on charter (no MOVE support). Seem OK on yahoo (has MOVE support). Just bad on gmail. I think the problem with gmail is that there are two different "capability" responses. The first one does *not* include MOVE but the 2nd one does. I *ass*umed that that the MOVE cap. would still be stored up in the 2nd capability response but apparently it isn't. Will continue looking...

Comment 6

5 months ago
Well, even on google I am not doing a call to UidExpunge() when moving. So it it must be seeing MOVE cap and doing a "real" move. However, something is still badly broken when trying to move or copy messages marked as deleted on gmail.
Maybe it's the same problem as exchange which I don't have. Is there a "public" exchange imap server I can try?

Comment 7

5 months ago
(In reply to Magnus Melin from comment #0)
> For some reason I hadn't noticed this before. Should be from bug 1231592.
> 
> Now when I move a message to another folder it's immediately removed and
> expunged. I'm using the mark-as-deleted imap model, so that's why it's
> showing, and it's unexpected.
> 
> And mail.server.default.forceSelect is false by default, so the behaviour
> should not have changed.
> 
> 
> Expected result: should not immediately expunge a message after move.

I assume this means it is expunged from the destination folder? The server should expunge it from the source folder when a MOVE command occurs. Right?

Comment 8

5 months ago
I asked that already ;-)

(In reply to Jorg K (GMT+2) from comment #3)
> Sure, we call UidExpunge(messageIdString), so just expunging the message
> that got moved on a server that didn't support MOVE, so had to be copied and
> might as well be expunged from the original source.

> Sure, for non-MOVE servers the behaviour has changed, but is it really not
> better now? You *move* something, so why keep a superseded copy in the old
> location?

So I'm not convinced that the new behaviour is worse then the old. You could just get Magnus to dump out whether the server has MOVE capability, but I guess from what we're seeing we can derive that it doesn't.

(In reply to gene smith from comment #6)
> ... on google I am not doing a call to UidExpunge() when moving. So it it must be
> seeing MOVE cap[ability] and doing a "real" move.
> However, something is still badly broken when trying to move or copy messages
> marked as deleted on gmail.
Why not focus on the bug here and do the Gmail stuff in another bug. So you're moving messages which have been marked as deleted with undesired effects? Note that there are a few problems with Gmail support, see for example bug 618553.

Comment 9

5 months ago
Hmm, what is the Gmail problem exactly, I'm doing this:
1) "Just mark it as deleted" in account settings.
2) Delete a bunch of message in folder A, they are now struck out.
3) Move them to folder B.
4) They disappear from folder A, good.
5) They appear in folder B as non-deleted, questionable, but not lethal.
6) Going back to folder A they don't reappear, good.

Comment 10

5 months ago
(In reply to Jorg K (GMT+2) from comment #9)
> Hmm, what is the Gmail problem exactly, I'm doing this:
> 1) "Just mark it as deleted" in account settings.
> 2) Delete a bunch of message in folder A, they are now struck out.
> 3) Move them to folder B.
> 4) They disappear from folder A, good.
> 5) They appear in folder B as non-deleted, questionable, but not lethal.
> 6) Going back to folder A they don't reappear, good.

At step 5 I don't see the moved messages at all! Is your folder B "All mail"? Mine's not.

I remember now when working with gmail when testing it was "different". Looking at the imap log, when you "delete" something, say in Inbox, tb marks it as /deleted. However, when the next fetch occurs on the folder, the /deleted message doesn't appear in the list sent back. However, it is still in the "All Mail" folder and not mark deleted, so even if expunged from other folders it is still really there. The only way you can really delete an email is move it to Trash and then empty Trash (or mark message in trash as deleted and then compact trash if in that mode).

So, in conclusion, I don't think there is not anything that can be done about gmail behavior. Also, I think you are saying that Magnus's only complaint is that the source message is expunged on a move but it displays OK in the destination (struck out). If so, then that is how it should work.

Comment 11

5 months ago
You have to be careful. Gmail uses labels,not IMAP folders, so the delete is simply removing the "delete" label from the email that is already resident in the all mail folder (the gmail archive of "all mail")

In gmail all mail is in the "all mail" folder.  that same mail can be in as many folders as it has labels attached.  Once the labels are all removed the mail still resides in the "all mail folder",  the archive.  

My understanding may be faulty,  but only when a delete is issued against the email in the "all mail" folder is it deleted.  But we then place it in the deleted folder and undo the delete in practice, because that adds it back to the all mail folder. So delete really does not work.  Or didn't. unless you held shift to bypass the delete folder.

Comment 12

5 months ago
Yes, I see that all gmail messages, Inbox, Sent, etc, are dumped into "All mail".
What I observer is that I can truly delete emails only if I MOVE them from "All mail" to Trash. I'm still testing in
"mark as deleted" mode and this leaves deleted emails X'd out in "All mail" and now present in Trash. Then in trash, I have to delete them again (they become X'd out) and then compact trash for them to go away in trash. In "all mail" they are still X'd out. I then compact "all mail" and they go away there too. If I restart tb and look again in "all mail" the delete messages are still gone. If I look on gmail app on tablet they also appear to be gone. So unless something is fooling me, the mails are actually deleted.
Doing shift-delete on an email in "all mail" X's it out but doesn't put it in trash. So a compact on "all mail" brings it right back and removes the X, so it doesn't get actually deleted.

Comment 13

5 months ago
Sorry, I had it in my head that Magnus was moving an email that had already been X'd out. 
Anyhow, signed up for an outlook.com account and it uses exchange server according to imap.log. Its CAPABILITY does not include MOVE and it supports UIDPLUS. So outlook does the simulated MOVE ending with UidExpunge() and what Magnus now sees is the new "expected". I see the same behavior with Yahoo (that supports MOVE) and with charter (that simulates move just like outlook -- no MOVE, yes UIDPLUS). Gmail also supports MOVE but it does not always expunge the MOVEed messages, depending on source folder, so you sometimes see X'd out messages remaining after they are moved.

Comment 14

5 months ago
Well, sorry, I'm wrong again in Comment 13. Gmail always expunges the moved emails in any source folder when I move emails NOT marked as deleted. I only see problems, as I point out in Comment 10, when I try to move messages that are already marked as deleted, i.e., X'd out. But Jorg says in Comment 9 he doesn't see this problem...? Then again, I don't think this is what Magnus was trying to do.
(Reporter)

Comment 15

5 months ago
I'll get back to the details later. But just to clarify, I'm moving a read, but not deleted message to another folder - and that message is then "instantly" removed from the source folder.
Flags: needinfo?(mkmelin+mozilla)

Comment 16

5 months ago
Yes, we understood that. But why is that bad? We all agree that the behaviour has changed, but why leave the X-ed message in the source folder?

Anyway, this behaviour can be restored easily by checking "Just mark it as deleted" before doing the UidExpunge(messageIdString).

Comment 17

5 months ago
(In reply to gene smith from comment #4)
> gmail's even worse. If you mark a message as deleted then move to another
> folder and then come back, the message goes away.
I didn't understand what you mean exactly here, hence my comment #9.

(In reply to gene smith from comment #10)
> At step 5 I don't see the moved messages at all! Is your folder B "All
> mail"? Mine's not.
I was moving messages between "normal folders".

> So, in conclusion, I don't think there is not anything that can be done
> about gmail behavior.
OK. Yes, let's not mix different (non-)issues in this bug. Let's close the discussion about Gmail.

> Also, I think you are saying that Magnus's only
> complaint is that the source message is expunged on a move ...
Yes, that is his complaint.

> If so, then that is how it should work.
I agree that if we can, and we can, we should *not* leave an X-ed message in the source folder.

So we have two options:
1) Magnus is wrong and the changed behaviour is better -> WONTFIX.
2) Magnus is right, so we do what I said in comment #16.

In any case, bug 1231592, didn't "break" anything. It changed the behaviour for non-MOVE servers in a well-understood way and we can either accept or revert that.
(Reporter)

Comment 18

5 months ago
CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=NTLM AUTH=GSSAPI SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+

... so no MOVE support.

(In reply to Jorg K (GMT+2) from comment #16)
> Yes, we understood that. But why is that bad? We all agree that the
> behaviour has changed, but why leave the X-ed message in the source folder?

That's the point of using the mark-as-deleted model. You get to delete/move messages quickly, and still have a super safe and easy way to get the message back if you changed your mind, or like it happened to me, worked too quickly and moved the message, but it had gone to the wrong folder so it was hard to find.

I do see it's simulating a MOVE. Just not sure I like it.
Flags: needinfo?(mkmelin+mozilla)

Comment 19

5 months ago
(In reply to Magnus Melin from comment #18)
> but it had gone to the wrong folder so it was hard to find.
That happens with local message, too ;-) In fact, undo works for IMAP, doesn't it?
And if you don't go and find the message, it will live in the wrong location forever.

> I do see it's simulating a MOVE. Just not sure I like it.
Well, someone has to make a decision. WONTFIX or do comment #16?
(Reporter)

Comment 20

5 months ago
(In reply to Jorg K (GMT+2) from comment #19)
> And if you don't go and find the message, it will live in the wrong location
> forever.

Yes but it's easier to find if you know the subject so you can just search for it!

> > I do see it's simulating a MOVE. Just not sure I like it.
> Well, someone has to make a decision. WONTFIX or do comment #16?

Let's see how easy it is to get used to it first.

Comment 21

5 months ago
On charter account (no MOVE support) and still in "just leave marked as deleted" mode, if you move messages to another folder the move can be reversed with menu Edit | Undo (or ctrl-z). It leaves the messages in the errant destination folder X'd out but they completely vanish if you compact the errant destination folder.

Comment 22

5 months ago
(In reply to Jorg K (GMT+2) from comment #8)
> Why not focus on the bug here and do the Gmail stuff in another bug.

Ok, here's my observation on outlook.com (exchange) and gmail: Bug 1353221
status-thunderbird55: --- → affected
Whiteboard: [regression:TB55]

Comment 23

4 months ago
(In reply to Magnus Melin from comment #0)
> Now when I move a message to another folder it's immediately removed and
> expunged.

Not exactly. RFC 6851 states that this is the intended behavior for the IMAP Move command:

| [...] the original message is removed from the source mailbox, and
| it appears to the client as a single action.  This has the same
| effect for each message as this sequence:
| 
| 1.  [UID] COPY
| 2.  [UID] STORE +FLAGS.SILENT \DELETED
| 3.  UID EXPUNGE

> I'm using the mark-as-deleted imap model, so that's why it's showing, and it's unexpected.

> Expected result: should not immediately expunge a message after move.

This would mean that the Move command can not be used in the case of mark-as-deleted.
(Reporter)

Comment 24

4 months ago
The behavior is correct (per spec) for a server supporting IMAP MOVE. The question is if that should behavior should be simulated for a server not supporting MOVE.
You need to log in before you can comment on or make changes to this bug.