288465 - Moving partially downloaded messages causes message to be lost.

Status: NEW

(MailNews Core :: Networking: POP, defect)

Description

[averk]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; ru-RU; rv:1.7.6) Gecko/20050226 Firefox/1.0.1
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru-RU; rv:1.7.6) Gecko/20050226 Firefox/1.0.1

Moving message another folder leaves link and allows to download the rest, but
after it orphan message from POP3 removed when you get messages next time, even
if folder is a subfolder of the same account. When you attempt to download such
message (stored in another folder), it disappers.

Worst addition to problem: when you trying to move partial messages from POP3
account to IMAP, IMAP gets an empty message with no download link and no updated
body. Orphan is deleted later as well.


Reproducible: Always

Steps to Reproduce:
1. Configure an account to receive only headers.
2. Get the messages and move it somewhere from inbox.
3. Get the messages again.

Actual Results:  
None of the messages is valid anymore, trying to click download link will remove
them without notice.

Expected Results:  
Keep track of moved messages and keep messages on server is partial message is
not deleted. When moving to IMAP, either download the message or mark it as
partial as well and track later.

Comment 1

[averk]

Bug still persists at 1.0.6, 100% repoducable

Comment 2

[David :Bienvenu]

re-opening. But have you tried the first part of this bug on 1.1a2?
ftp://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/1.1a2

Comment 3

[averk]

(In reply to comment #2)
> re-opening. But have you tried the first part of this bug on 1.1a2?
> ftp://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/1.1a2

Yes, works absolutely the same. Message is deleted from server after next POP3
receive session, then, after attempt to get it, it disappears from message list
as well.

Comment 4

[David :Bienvenu]

are you limiting the message size of automatically downloaded pop3 messages, or
are you downloading headers only? Do you have tbird set to automatically delete
from server when messages are moved or deleted?

Comment 5

[averk]

(In reply to comment #4)
> are you limiting the message size of automatically downloaded pop3 messages, or
> are you downloading headers only? Do you have tbird set to automatically delete
> from server when messages are moved or deleted?

No limits, no trash emptying, no disk space saving - everything is almost by
default, except Server Settings -> Server Settings, where only "Fetch headers
only" is checked, and I even don't know about existence of a special option to
"automatically delete from server when messages are moved or deleted".

Comment 6

[David :Bienvenu]

ah, so you are fetching headers only. That could be a factor...Can you e-mail me
 your prefs.js so I can be sure what your settings are?

Comment 7

[averk]

Version 2 beta 2: moving to local folders is now safe. However, when you try to move it to IMAP folder, it's neither downloaded automatically nor left as is: message body appears empty, so you can't download the rest of message anymore. If such message is deleted later, it's "ghost" remains at server and can be retreived  or deleted from mailbox only by other client or webmail.

Reproducible: Always

Steps to Reproduce:
1. Configure POP3 account to retreive headers only.
2. Get new messages, but don't download them fully. 
3. Move message to any folder at IMAP account.

Actual Results:
Message at IMAP folder looks empty and can't be downloaded anymore. When you delete it, actual message remains at POP3 mailbox, but ignored by Thunderbird at send/receive.

Expected Results:
Either download the message or mark it as partial same way as at local folders and track it later with ability to download and remove.

Generally, bug is no more critical, such actions do not lead to unconditional data loss anymore.

Comment 8

[Daniel Ferber]

I could reproduce this bug in another scenario.

I configured TB to not to load messages over 500KB (POP account). When I receive a large message, it displays an information that the message was partially loaded and that I have to click on the link to load the complete message. This is working as expected.

I also have the anti-spam enabled. When a message is identified as spam, the message is moved to the "spam" folder. This is working as expected.

The account is configured to remove the message from the server if removed locally from the "inbox". When the anti-spam moves a message from the "inbox" to the "spam" folder, TB deletes the message from the server. This is working as expected.

Now the (possible) bug: I receive a large message that is identified (erroneously) aby TB s spam. The message is partially received, because it is large. The message is moved to the "spam" folder and removed from "inbox" and therefore removed from from the server. The user finds the message in the "spam" folder and sees that the message is not spam. The user moves the message back to "inbox". Then, in order to read the message, the user will click to receive the complete message, but it does not exist anymore on the server. TB simply removes the message from the "inbox" and the user thinks that the message had simply disappeared.

The problem is that TB should only remove a message from the server if it was removed from the "inbox" AND completely received. Or TB should download the entire message when moving to another folder... There may be other better criteria... But TB should not loose messages mysteriously.

Comment 9

[Wayne Mery (:wsmwk)]

Can your reproduce this with beta version?
  http://www.mozillamessaging.com/en-US/thunderbird/early_releases/

backup your profile first

Comment 10

[averk]

Bug is 100% reproducible according to steps at message #7.
To make a report more precise - this happens only if you move a message to IMAP folder, local and POP3 account folders allow to download the rest of the message correctly. 
Expected behaviour is either to use IMAP the same way, to download the message automatically before moving or to restrict moving until the message is downloaded manually.

Comment 11

[Wayne Mery (:wsmwk)]

averk, if you can still reproduce this in version 5, then I am pretty sure there are newer pop bugs filed on the same issue

Comment 12

[averk]

Same story, as at message #7 (and, subsequently, #10). Bug is still there at version 5. Sorry for delay.

Comment 13

[Wayne Mery (:wsmwk)]

averk, thanks for the update. which of these bugs is a maching duplicate?

bug 287728 Messages lost with fetch headers only + leave on server until moved + junk filter
bug 360300 	'fetch headers only' mixed with 'until i delete or move them from inbox' can result in lost message bodies
bug 541990 Moved junk on POP3 accounts with headers-only stays on the server

Comment 14

[averk]

bug 541990 is definitely a copy of the current state of this bug;

bug 287728 is originally a duplicate, but with an extension at comment 21 (duplicate a partial message and lose second copy). According to my tests, it works exactly, as stated originally at comment #21, and doesn't crash, as at comment #23;

bug 360300 is also a duplicate of a solved part of this bug, except of comment #6, which is unrelated to the original bug, and seem not be a bug at all;

And one more addition: account with orphan messages left after IMAP manipulations reports new mail at POP account tree item (obviously, unread orphan mail) at automatic checks (only at automatic check, not when you check it yourself!), but can't retreive, count it, or report a folder, where it's located. When you check mail, account state is cleared until next automatic check starts.

My suggestion is to enforce download when you make a copy of the original message or move it to another account.

Comment 15

[Wayne Mery (:wsmwk)]

ref: comment 14

Comment 16

[:aceman]

There is also bug 714090.

Comment 17

[:aceman]

There is a discussion that this happens intentionally in bug 645976.

Comment 18

[:aceman]

Comment 8 seems to be 287728.

OK, it seems today only the case of moving the partial message to IMAP is left here, comment 7.

We could argue if that is actually a bug. You decided to move a partial message to an IMAP server. We can't track the message indefinitely across servers (maybe we could but it would be a ton of work and maybe still buggy). So only the partial body was stored on the IMAP server. The message should be removed from the POP3 server (the partial version locally and also the full version on server). If you wanted the full copy, you should have downloaded it before (some users want the partial copy).

Can we refine this in some way? Prevent partial messages to be copied out of their original account? I don't think so. Some users may want to only store the headers (e.g. subjects). Should we warn the user when copying that msgs is partial? Other ideas?

Comment 19

[Wayne Mery (:wsmwk)]

(In reply to :aceman from comment #18)
> Should we warn the user when copying that msgs is partial? Other ideas?

That sounds like a start. Perhaps even the best we can do?

Comment 20

[Wayne Mery (:wsmwk)]

Bug 360300 is a dupe?

Comment 21

[Magnus Melin [:mkmelin]]

Probably the best we can do is "to enforce download when you make a copy of the original message or move it to another account.".

Comment 22

[averk]

(In reply to Wayne Mery (:wsmwk) from comment #20)
> Bug 360300 is a dupe?
Even if it's not a dupe, it seems to have the same reason.

Comment 23

[averk]

(In reply to Magnus Melin from comment #21)
> Probably the best we can do is "to enforce download when you make a copy of
> the original message or move it to another account.".

If we don't want to celebrate the next anniversary of this bug, this would be the most simple and logical solution: unless we move message to a trash can, we don't want to lose it and, mostly like, plan to download it later. So, finally, it not a serious contradiction to user's wishes.

An option to keep tracking this message between accounts would require additional checks at account (or even folder) rename/remove operations, and leave numerous bugs as a heritage, so it's not worth doing.

Anyway, it's a data loss bug, so any reasonable and consistent way to close it would be a victory. 
I think that enforcement matches these considerations fully. It could also eliminate a cause of several related bugs mentioned here and at #360300.

Comment 24

[:aceman]

I don't like that solution:
1. there was a reason the user only downloaded partial messages. Maybe it was saving data transfers. Forcing downloading of all messages would defeat that point and may produce problems for the user.
2. this would be the hardest solution of them all. Starting POP3 downloads from inside the copy routines would be hell to code and would make this bug drag for another 10 years.
3. also even if it worked, it would be totally slow. Yes, we could warn the user about it. But if something failed in the process (server timeout, connection loss), then what? Abort whole copy? Copy only sucessfully fetched messages? Either there would be many questions for the user or the logic would become quite complicated and would open lots of UX discussions.

My proposed solution would be just to warn the user that there are partial messages whose bodies would be lost (we can be conservative and say they just will be lost, even if maybe sometimes it would be true (like if he put them back into Inbox)). If the user confirms it, the messages would be copied normally. I would even go as far as to remove the "partial" flag from them so next time there is no warning. The problem with this would be that the messages would appears as complete since then, while the body wouldn't really be complete. Which could confuse the user. We could harcode a reminder into the message body, that this is an incomplete message. But not with the automatically generated link as today, but a hardcoded string into the message body. Similarly like we detach attachments now. Unfortunatelly that procedure is quite complicated again.