1528297 - Message prematurely deleted on POP server (via "d" in popstate.dat) if an attachment is deleted or detached in TB (locally creates new message & deletes the original, which triggers server deletion with "Leave messages on server until I delete them")

Status: RESOLVED FIXED

(MailNews Core :: Networking: POP, defect, P2)

Description

[Laurent Grenet]

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0

Steps to reproduce:

The server is a POP server.
Option is "Leave message on server" during 14 days or until they are suppressed locally

Receive from server a mail with attachments
Locally, delete or detach attachments, but DO NOT delete the message itself

Actual results:

At next connection to server, message is fully deleted from server, while it shouldn't

Expected results:

Message should remain on server. No matter it remains with its attachment, or without them, but message itself shouldn't be deleted on server since

• less than 14 days from receiving
• message NOT deleted locally

A candidate cause of this erroneous behaviour is the fact that every time attachments are detached or deleted from a message, the corresponding message Id in popsate.dat is prefixed with a 'd' rather than the original 'k'

Comment 1

[Jorg K (CEST = GMT+2)]

This is ugly. Deleting/detaching an attachment will delete the message internally and create a new one without the attachment(s), but that shouldn't lead to deletion on the server. Ouch.

Comment 2

[Laurent Grenet]

I'm surprised this bug is still not fixed... while

• it is easy to reproduce (and I just did a test to verify it is still there....)
• and is easily understandable :

I just ran a "5 minutes investigation", and I saw that when the original message is deleted, its reference in popstate.dat is prefixed by a "d" (as delete ?) rather than a "k" just before... but when the new one (without the attachment) is created in place of original one, popstate.dat is not updated...

As a result, consequence on popstate.dat file is the same either if we delete attachment or if we delete the message itself.
It is then "normal" that consequence on message on the server is also the same....

This bug should then be quite easy to fix, isn't it ?

Comment 3

[Laurent Grenet]

I just missed to mention that the bug is still there in TB 78.2.2

Comment 4

[sajent]

I'm using 78.8.1 on an IMAP server. Deleting an attachment from a message in the Sent folder deletes the message, and I don't then find it in the Trash folder like messages that I've deleted. It's just gone.

Messages in the Inbox folder don't similarly disappear if I delete an attachment.

Comment 5

[Wayne Mery (:wsmwk)]

Do you still see this when using version 102 ?

Comment 6

[sajent]

On 102.2.2 I no longer see this failure.

Comment 7

[Dan Pernokis]

I've never seen this. Just tested on TB 91.13.0 for reference -- works OK, nothing deleted on server. Then on TB 102.2.2 -- ditto, nothing deleted on server.

I sent identifiable pairs of messages to myself and deleted the attachment from one of the pair. All messages remain present and intact on the server -- no deletions, no attachments removed either.

Opinion...
As for deleting just the attachment on the Server when deleting it locally (see comment in STR), I would strongly disagree with this design, and I pray nobody would think to implement it! The reason you don't delete from the Server is because you may want to re-fetch it, or re-view it, or -- in my case -- poll it down to multiple devices. If Device A deletes an attachment, Device B will never get to see it.

Comment 8

[Laurent Grenet]

Sorry for late answer, but I was far from home when receiving the request.

But, unfortunately, and contrary to what others answered above, THE BUG IS STILL PRESENT in TB 102.2.2 and NOT FIXED AT ALL.

The answers above seem to mean that the bug is not correctly understood. So let me restate it once more.
First of all, I'm not talking about IMAP, but POP3 (as mentioned in my initial post...)

1. I receive, in POP3, a mail including an attached file. Since my POP options are "Leave message on server" during 14 days or until they are suppressed locally, the message remains on the server.
2. Locally, in TB client, I delete, or I detach the attachment.
   At this step, message still remain on the server. It is NOT attachment deletion that generate immediately mail deletion on server.
3. But, as soon as I request again for new mails from the server, and so as soon as local TB synchronize with server, the mail IS DELETED on the server.

This is "as if" TB considered that the mail itself (and not only its attachment) were deleted (what IS NOT the case), and then apply (erroneously in this case) the option "Leave message on server" during 14 days OR UNTIL THEY ARE SUPPRESSED LOCALLY

Comment 9

[Dan Pernokis]

Let's try Comment 7 all over again...

I can confirm the bug as clarified by the reporter in Comment 8 does still exist and persist. The misunderstanding was in the "next connection to server" -- I took that to mean next time Webmail was checked, whereas the reporter actually meant next time TB re-connected & polled.

To be clear, I am running POP3 accounts. But I had not checked the "Or Until I Delete Them" boxes on the accounts** -- that's what the English-language version says. (FYI, I've never used that setting -- I poll from multiple devices, and I don't want Device A deleting messages locally and causing them to disappear from the Server before Device B polls them.) For this new set of tests, I did check the box on the accounts.

I first tested on TB 91.13.0 for reference, especially since this bug pre-dates TB 102. Then I tried TB 102.2.2 official release and compared results.

TESTING TB 91.13.0...
I sent identifiable pairs of messages to myself, downloaded them, and checked Webmail -- all were on the server intact -- then logged out of Webmail. Then I deleted the attachment from one of each pair, closed TB, disconnected from Internet, reconnected, and re-checked Webmail. All messages were still present & intact, and I logged out (to avoid interference). I re-launched TB, polled for messages (nothing new, none came down), then checked Webmail again. Still, all messages were present & intact.

TESTING TB 102.2.2...
Ditto - same results exactly.

BUT...
On a whim, I decided to try another of my accounts known to be on a different mail server (at a different ISP). I performed the same steps as above, but suddenly messages would disappear from Webmail whenever I deleted an attachment locally, closed & re-launched TB, and polled the account for new messages -- exactly as the reporter claimed! Definitely a bug! Deleting an attachment locally should not affect the mail server copy (nor should it delete the attachment on the server -- unless that function is implemented as an option requiring positive action by the user to turn it on.

As a work-around (and the reporter probably does this):
Uncheck the "Or until I delete them" box on any relevant accounts. Since the delete time is only 14 days in this case, the messages will disappear soon enough anyway. For the rare occasions when full deletion is required from the server, it can be done manually in Webmail.

Comment 10

[Dan Pernokis]

Further to my Comment 9...
Definitely a bug in that TB must be sending a deletion command of some kind to the mail server -- possibly to delete just the attachment.

But looks like servers interpret such a command differently: (i) some as a deletion of the complete message; (ii) some simply ignore it; and (iii) perhaps some servers delete just the attachment as intended -- although I haven't seen this in my limited scope. (But I wouldn't want that functionality anyway unless it was fully optional for the user -- we can't have specific users deleting attachments in a multi-user environment before they have been seen...)

Comment 11

[Laurent Grenet]

Since Dan said that he reproduced the bug with some mail servers and not with some others, I did the test with several mail servers.

And I confirm that the bug exists

• with Gmail (Pop server = pop.gmail.com / webmail = https://mail.google.com),
• and also with free.fr (Pop server = pop.free.fr / webmail = https://zimbra.free.fr)
• but on the other hand, do not exist with laposte.net (Pop server = pop.laposte.net / webmail = https://web-mail.laposte.net)

I also had a check to the "Activity manager" window of TB.
And I can see that each time I remove (locally) an attachment, TWO "activities" are logged in the activity manager

• Moved 1 message from Inbox to Inbox
• Deleted 1 message from Inbox

It looks like that each time an attachment is removed (locally), TB DOES NOT UPDATE the message, but actually DELETES the original message and CREATES a new one, without the attachment. If my assumption is right, this "could" explain deletion on server at new synchronization, if original message has been actually deleted...

Comment 12

[Laurent Grenet]

Just one complementary comment to my comment #11 just above :

Actually, EVEN on laposte.net mail server, WE DO HAVE the problem.
During my previous test, it was a "false correct behaviour", due to the fact that, at test time, any deletion was impossible on mail server, even in webmail... I don't know the reasons (most likely a temporary blocking situation on the server), but new test run a few hours later shows that both

• mail voluntarily deleted locally
• and mail not deleted, but having only their attachment deleted
  are BOTH deleted on the server a next synchronization (next mail poll)

As a result, according my investigations, BUG IS PRESENT ON ALL MAILS SERVERS I have a account on.

Comment 13

[Dan Pernokis]

I tend to agree with Laurent's conclusion in his Comment 11 that the mechanism of removing an attachment -- making a new version and deleting the old -- probably triggers deletion on the server. The new copy is created because how else would TB delete one attachment of many if it is in the middle without leaving a "hole" in the message? Then the bug appears whereby the deletion of the old copy INADVERTENTLY sends a delete code to the server (as though this was a full & proper deletion). TB should just do its copy part without sending a deletion code to the server.

What I don't quite get is why does it take until the next "sync" or contact from TB before the deletion occurs on the server? In my testing, the message remained intact until TB was relaunched. And then why do some servers -- one of my ISPs, for example -- not delete at all? There would be no difference to these servers in seeing a delete code based on a random (on-demand) local user-triggered delete and a local deletion based on expiry of the days. TB still has to send the delete code, and the server still has to recognize it, and yet timed deletes work fine everywhere for me. How are they different? (Presuming my first statement above is true...)

Comment 14

[Laurent Grenet]

I'm not sure that we can write that local action (delete only attachment, or delete completely a mail) "triggers deletion on the server".
It is not exactly "triggering", since it's not done immediately, but it is delayed until next polling of new mails from the server.

But anyway, how is it managed : very simply, thanks to the popstate.dat file managed by TB in the local folder corresponding to each mail server accessed in POP (the popstate.dat file is there, even if we have chosen to store mails themselves in another folder). It is this popstate.dat that "informs" TB that some mails are already there (locally) and do not have to be downloaded again (if you delete this file, you receive again all mails present on the server, whatever you already downloaded them or not)

And what can we see in this file :
Initial situation : my mail box contains 2 mails each with an attachment, plus a 3rd one we won't touch in the test, and here is popstate.dat's content
(I don't know what initial k stands for, then we have mail id given by the server, and then date of receiving in Unix timestamp format)

# POP3 State File
# This is a generated file!  Do not edit.

*pop.gmail.com recent:[e-mail]
k GmailId182ff5d32f594d51 1663085270
k GmailId1833799f2e779dc2 1663085271
k GmailId183379a89548668f 1663085271

Then I delete attachment of first mail, and popstate.dat becomes :

# POP3 State File
# This is a generated file!  Do not edit.

*pop.gmail.com recent:[e-mail]
k GmailId182ff5d32f594d51 1663085270
d GmailId1833799f2e779dc2 1663085271
k GmailId183379a89548668f 1663085271

Then I delete completely the second mail

# POP3 State File
# This is a generated file!  Do not edit.

*pop.gmail.com recent:[e-mail]
k GmailId182ff5d32f594d51 1663085270
d GmailId1833799f2e779dc2 1663085271
d GmailId183379a89548668f 1663085271

Eventually, I synchronize with the server (ie. poll for new mails).

# POP3 State File
# This is a generated file!  Do not edit.

*pop.gmail.com recent:[e-mail]
k GmailId182ff5d32f594d51 1663085270

As said above, I don't know what exactly means the k and the d that prefix each line (even if I bet for d = delete....), but if TB marks the same way in popstate.dat mails deleted and mails not deleted but having "only" their attachment deleted, it's not surprising that command sent to the server is the same, ie. a delete in both cases....

This is what I already tried to explain in my comment #2 two years ago....
But it was maybe not so clear, and it's the reason why I tried to re-explain with more details....

I hope this will be useful for people skilled enough to fix the bug....

Comment 15

[Laurent Grenet]

In complement to my comment #14 above, I've found more explanations about popstate.dat here :
http://kb.mozillazine.org/Popstate.dat

Comment 16

[Dan Pernokis]

Just for clarity in Comment 13 and Comment 14...
"Trigger" sets some process in motion -- can be immediately (as in delete locally and the delete happens right away on the server) or can be a series of steps, eventually leading to deletion on the server (as in delete locally, do-what-you-do, re-sync, etc, etc, then delete on the server).

And apologies for missing your Comment 2. It is clear once we read the rest of the comments, but on its own it's easy to miss the little nuances if the reader -- me in this case -- is not fully familiar with the internals of the process.

Comment 17

[Dan Pernokis]

Thanks for the description of POPSTATE, Laurent -- now I understand much better.

In my Comment 13, I said:

The bug appears whereby the deletion of the old copy INADVERTENTLY sends a delete code to the server (as though this was a full & proper deletion). TB should just do its copy part without sending a deletion code to the server."

Your POPSTATE file shows the message entry now tagged "d" which means DELETE (according to the doc -- read to the end, the "d" & "k" codes are described there). That's the bug, and the INADVERTENT part. TB should NOT change the status to "d" in the circumstance where only an attachment is involved. TB should create the copy in whatever way it needs to do to omit (get rid of) the attachment -- but it should NOT be writing out the "d" status to POPSTATE! (Writing "d" status explains the full deletion, and explains to me why the delete waits for the interaction of TB polling the server to fulfill it.)

And just for good measure, I also have to suggest that the status should remain unchanged -- presumably at "k", since "b" or "f" couldn't have happened yet (we're dealing with a message already fully downloaded with attachments). Perhaps flip to "K" if necessary to indicate a deleted attachment, but preserve the original "k"-ness of the status.

Comment 18

[Thomas D. (:thomas8)]

Ping, maybe when you have time... ?

Confirmed as still happening in TB 102 in comment 9. Imo this deserves attention as this involves real dataloss of messages including their attachments on server for affected users. This is POP, so deleting/removing attachments from the local copy of the message should not affect the server copy at all!

Maybe easy to fix if we just stop marking the message with "d" for deletion in popstate.dat when the user manipulates only the local copy of the message (which will delete the local copy of the message and create a new local message without attachments, but it should not mark the server copy of the message for deletion in popstate.dat). Note that the message might already be marked with "d" for deletion plus a timestamp due to the "Leave messages on server for at most 14 days" account pref. We should just not touch popstate.dat when removing attachments of the local message (delete/detach attachment).

There's very detailed analysis on the bug and hints for fixing this (e.g. comment 17). popstate.dat is explained here: http://kb.mozillazine.org/Popstate.dat

Comment 19

[Thomas D. (:thomas8)]

Note:

• This bug requires that "Leave messages on server until I delete them" is set (which is default setting).
• We delete the original local message and recreate a new one without attachments.
• Deleting the original message then triggers "mark for deletion on server" per the above pref (but should not in this case, so we probably need to distinguish plain vanilla message deletion from "delete for recreation w/o attachments" here).
• As long as things work like this, fixing this bug will then mean that manually deleting the new, trimmed local message will not effect deletion of the full original message on server any more, somewhat contrary to expectation. I guess that's acceptable because first deleting only attachments, then later deleting the rest of the message might be an unlikely scenario (should be easier to just archive the entire message in your file system then, or to move it to local folders, assuming that triggers server deletion too). It's definitely better to err in favor of data preservation.

Comment 20

[Ping Chen [:rnons] (away)]

Attachment: Bug 1528297 - Prevent deleting POP3 message on server after detaching attachments. r=mkmelin

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/9184df231824
Prevent deleting POP3 message on server after detaching attachments. r=mkmelin