287728 - Messages lost with pop fetch headers only + leave on server until moved + junk filter

Status: RESOLVED WORKSFORME

(MailNews Core :: Filters, defect)

Description

[Taliver Heath]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.6) Gecko/20050301 MultiZilla/1.6.3.1d (Debian package 1.0.1-1)
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.6) Gecko/20050301 MultiZilla/1.6.3.1d (Debian package 1.0.1-1)

If you choose the following preferences, your email will disappear:
Leave messages on Server... until I move or delete them from the inbox
and
Fetch Headers only.

When the junk mail filter detects a message as junk, it will move it to the junk
folder.  This will then delete it from the server without downloading the body.
 When you see the message in the junk mail and attempt to fetch it, the mail
will go away completely.

Reproducible: Always

Steps to Reproduce:
1.Set Fetch Headers Only
2.Set Delete from server when I move mail from inbox
3. Wait for a junk mail

Actual Results:  
Junk mail subject line can be seen.  But you cannot download the message. 
Downloading it will make the entire email disappear.

Expected Results:  
1) Since the junk mail filter does its work automatically, junk mail should not
be deleted from the server until it is deleted from the junk mail folder.  
2) Messages that are deleted from the server but whose subjects you still have
should display a message in the body, such as "mail deleted from server", and
leave the subject/from/to, so that a request for a resend from the person in
question can be done.

Comment 1

[Tuukka Tolvanen (sp3000)]

I couldn't reproduce the bug with thunderbird trunk 20050313 on linux.
(Tweaking summary; old: Preference choice creates disappearing email)
Reporter, can you test a trunk nightly build? (Keep backups :)

Comment 2

[Tuukka Tolvanen (sp3000)]

Hmmm. Now I did see one junk message disappear when clicking the 'here' link to
download it. I think that message didn't get filtered quite on fetch in the
first place, but later on, for some reason. Might have been some other setting
set differently at that time.

Comment 3

[Taliver Heath]

I'm using the 1.0.2 release from debian, and haven't tested these issues against
nightly builds.  Should I?

Comment 4

[Albert Feller]

(In reply to comment #3)
> I'm using the 1.0.2 release from debian, and haven't tested these issues against
> nightly builds.  Should I?

yes! it is always good to reproduce the bug with a recent nightly/a fresh profile.

Comment 5

[Oleg Abrosimov]

C'mon folks! The last comment on this bug is as old as 2005-03-26. Now is 23.01.2006, I'm using TB 1.5 and I've lost ALL my VERY IMPORTANT e-mails from POP3 server! Damn!!!!

Ok, settle down and re-start the discussion.
To reproduce this bug:
0) create new POP3 account
1) in Account preferences check Leave messages on server + Until I delete or move them from Inbox + Fetch headers only
2) in Junk Mail Controls check Move incoming messages determined as junk mail to (Junk folder on:...) + Under Adaptive Filter tab check Enable adaptive junk mail detection checkbox

Congradulations! Now you are ready to loose all your mail! Just click on get mail button. headers will be loaded. Then, under Tools menu select "Run Junk mail controls on folder", - and all! messages will be marked as junk, because adaptive junk filters are stupid for now. All message headers will be moved to Junk folder and it is a time for "Until I delete or move them from Inbox" option. As a result - all messages will be deleted from server without any chance to recover. And yes, if you want to load some message in Junk folder it will just dissapear because it is not exists.

The problem is in "Until I delete or move them from Inbox" option. It is completely wrong.
1) I can move it from inbox to any other folder (e.g. My Friends) and even not "I", but my automatic mail filters. and again - messages will dissapear.
Solution: replace it with "Until I delete them"

second minor problem with junk filtering:
after automatically marking all messages as junk and moving it to Junk folder, as described above, it is not so simple to unmark and move them back to inbox.
consider: to mark message as junk and move it to Junk folder I need just one click to mark it as junk and it will be moved automatically, but if I'm in a Junk folder, and the same click to unmark just unmarks it and not moves to inbox. This behavior is inconsistent.

Comment 6

[Mike Cowperthwaite]

xref bug 248742

Comment 7

[Kathleen]

It is now 1/10/2007 and Thunderbird (1.5.0.9) is STILL DOING THIS!

please fix it!

Comment 9

[Mike Cowperthwaite]

Is this still reproducible with TB 2.0?

Comment 10

[(not reading, please use [e-mail] instead)]

sorry for the spam.  making bugzilla reflect reality as I'm not working on these bugs.  filter on FOOBARCHEESE to remove these in bulk.

Comment 11

[Kathleen]

(In reply to comment #9)
> Is this still reproducible with TB 2.0?
> 

This bug doesn't really apply any more, as step #2 no longer exists.

(2.Set Delete from server when I move mail from inbox)

With the new choices in TBird 2, it's turned into https://bugzilla.mozilla.org/show_bug.cgi?id=385127, which quite minor.

Comment 12

[Wayne Mery (:wsmwk)]

Kathleen can you dupe your bug 340010 to this and change OS=ALL?

Comment 13

[Olaf Schreiner]

(In reply to comment #11)
>
> This bug doesn't really apply any more, as step #2 no longer exists.

Well, I have to go ahead and disagree with you there. I am using version 2.0.0.6 build 20070728 (linux) and I am experiencing the same problem. I am using a POP account with the options (translated from German):

-leave messages on server / until I delete them
-fetch headers only

Additionally, I have set up a filter rule that moves certain mails from Inbox to another folder on delivery.

When I click to download the message body (link in body window) of one of those moved mails, the message disappears immediately (actual result, always). Expected result would be: message body appears.

This behavior rather seems to be a design flaw than an error in intended functionality. I suspect the following:
Moving an email out of Inbox triggers the option "until I delete them", because that particular message disappears from the local inbox and is therefore deleted from the POP server. Same thing if executed by filter rule.
Since the message is gone from the server as soon as the moved mail arrives in its destination folder, downloading the body must fail.

There are two ways out of this problem:
1.) Forcing downloading the message body when moving mail out of Inbox.  -or-
2.) keeping a message in the remote inbox until the last "headers-only" instance of that message disappears locally (which includes turning headers-only instances into full-message-instances)

Solution (1) is way easier to implement and therefore probably preferable, although the behavior of solution (2) might be more desirable.

Solution (2) comes along with a whole bunch of problems: If you keep a copy of the moved mail in the remote inbox, then the remote inbox and the local inbox must be different, meaning the mail must be hidden in local inbox. Secondly you need to keep track of headers-only instances of mails, which might become complicated as soon as they get copied around local folders. Or else local copying must be disabled until the body is downloaded.

Comment 16

[Mike Cowperthwaite]

(In reply to comment #13)
> When I click to download the message body (link in body window) of one of
> those moved mails, the message disappears immediately (actual result,
> always).  Expected result would be: message body appears.

Just to be sure: you're seeing the message disappear on the computer from which you directed Download Message, right?  In addition to deletion-from-server, which would affect downloads from other computers.


> This behavior rather seems to be a design flaw than an error in intended
> functionality. I suspect the following:
> Moving an email out of Inbox triggers the option "until I delete them",
> because that particular message disappears from the local inbox and is
> therefore deleted from the POP server. Same thing if executed by filter rule.
> Since the message is gone from the server as soon as the moved mail arrives
> in its destination folder, downloading the body must fail.

This was the change for 2.0 that prompted comment 9:  moving the message from the Inbox no longer triggers deletion from server.  I just tested a headers-
only message with the filter case and it worked as expected: from the new folder, I clicked Download Message and it showed up.  The Junk case is harder 
to test, since I don't receive much junk, but I'll keep my eyes on it.


> There are two ways out of this problem:
> 1.) Forcing downloading the message body when moving mail out of Inbox.

Which you can do as a filter action, for messages that are moved by filters.


> 2.) keeping a message in the remote inbox until the last "headers-only"
> instance of that message disappears locally (which includes turning
> headers-only instances into full-message-instances)

Maybe I'm missing something here.  What is the scenario involving multiple "headers-only" instances?

Comment 17

[Olaf Schreiner]

After I had abandoned the discussion for quite awhile (and not used the headers-only option) I tried to reproduce the problem again with version 2.0.0.19 (20081209) linux, which I am currently using and the problem seems to be gone. This is what I did:

- check headers-only option
- check remove from server when removed from inbox
- create filter which moves incoming mails with "filter-test" in subject to waste folder
- send mail to myself with subject "filter-test"
- go to waste folder, select the received mail
- click on link in body window that downloads actual message body

Effect: everything works fine -- the body appears.
I suggest, that this bug be closed.

Thanks a lot to everybody who helped solve the bug.

Comment 18

[Olaf Schreiner]

(In reply to comment #17)

I did the same procedure in 3.0b1 (no build info given in About) linux. It worked fine as well.

Comment 19

[Wayne Mery (:wsmwk)]

Olaf, thanks for the test. Do you agree bug 248742 is the same?  If so we can d mark 248742 WFM and dupe this to it.  Thoughts on bug 249452?

Taliver, the reporter, appears to be gone,   Kathleen's address is dead, and mcow in bug 248742 comment 13 said he is unlikely to reproduce the problem anymore, so I'm not sure we're going to hear from anyone else.

Comment 20

[Olaf Schreiner]

(In reply to comment #19)
> Do you agree bug 248742 is the same?

It clearly seems to be related. As far as I understand, the difference is that in 248742 the leave-on-server option is unused, which may or may not make for a different cause.

I tried to reproduce the bug in the same way as in comment #17 of this thread, except that I disabled leave-on-server beforehand. Everything worked as expected. I was not able to reproduce the error.

It seems likely, that the bug (248742) is not there anymore. It could very well be that it had the same cause. I suggest further testing, though.

Comment 21

[Olaf Schreiner]

(In reply to comment #19)
> Thoughts on bug 249452?

I reproduced it (bug 249452) in 2.0.0.19 linux, and yes the problem is still there. Here is how to reproduce it:

- check headers-only option
- uncheck leave-on-server option
- send yourself an email
- copy the received email to another folder
- click download body in first copy
- click download body in second copy

BOOM: the second message copy will disappear.

It is not a bug, though, but rather a design flaw, as I elaborated in #11 of this thread already: If the user makes multiple copies of a headers-only message, he/she will expect, that the message body can be downloaded for each of those copies. However, if leave-on-server is unused, the body download of the first message-copy will delete the message on the server.  Body download will then be unavailable for all other copies of that headers-only message.

I assume, it is irrelevant, whether these copies of the headers-only message are made manually or by a filter.

I see five ways out of this misbehavior:

1. force body download on message copying (automatically, before the actual copy is created).

2. keep track of how many copies there are and delete the message on the server only after the body has been downloaded for all existing copies.

3. create links instead of real copies (or another mechanism which allows, that for all copies of a message, the message is really only stored once).

4. stick with multiple instances for copies (as is), but perform body download for all copies at once, when triggered for one of them.

5. stick with everything as is, except: when a message body is downloaded for the first copy of a headers-only message, mark this message to be a mirror of the original download location. For all other copies of the message, "download" the body from this mirror.

The advantage of solution 1 is that it is the easiest to implement. But users might be confused about the automatic downloading.

Solution 3 is the cleanest: it solves the problem and saves local storage space on top of it. On the other hand it may not be easy to implement. Also, it may not be supported by the folder storage format and those folders might then not be usable by other application (minor concern).

Solution 2 is not pretty, but is a way to implement the expected behavior in the given circumstances (file formats etc.). It will be hard to ensure consistency here.

Solution 4 is merely a variation of solution 2 (implementation-wise), because in both cases one has to keep track of which messages are copies of which.

Solution 5 is probably the most practical. Everything can stay as is. It will be ok to delete the remote message on the first body download. Nobody needs to know, which messages are copies of which.  The only problem, which has to be solved, is how to find the local mirror message, when the body is requested by a later copy.
Oh, and there is one more problem with this solution: what if the mirror message gets deleted, before other messages request the body...

OK, enough said.

Comment 22

[Wayne Mery (:wsmwk)]

similar bugs:
bug 288465
bug 360300
bug 274050

Comment 23

[Rob]

Confurmed in 3.1.6. Currently, this actually crashes thunderbird :) I tried the steps in comment 21 and upon trying to download the message for the second time (in the copy), thunderbird crashed.

Comment 24

[Wayne Mery (:wsmwk)]

Preetham, Meharli, can, you work up a solution?

Comment 25

[Parkhideh]

Thunderbird 16.0.2;  Windows OS
Wayne,  please see bug 360300  comment 12; I think the option UNTIL I DELETE THEM is broken.

Comment 27

[Ping Chen [:rnons] (away)]

Seems to me 'until i delete or move them from inbox' has become 'until i delete them'. And move a message doesn't remove it from the server.