571693 - Data loss/file corruption of incoming pop mail starting 20100609 trunk nightlies (TB and SM)

Status: RESOLVED DUPLICATE of bug 582918

(MailNews Core :: Networking: POP, defect)

Description

[Daniel Kies]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.3a5pre) Gecko/20100608 Lightning/1.1a1pre SeaMonkey/2.1a2pre (Like Firefox/3.6)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.3a5pre) Gecko/20100608 Lightning/1.1a1pre SeaMonkey/2.1a2pre

Starting with 20100609 nightlies of both TB and SM, mail is lost when downloading new mail.  The status line reads "Downloading n of nn messages" but only a fraction of the new messages appear.  Rebuilding the folder results in loss of data, and mysteriously the reappearance of old email previous lost in those builds. 

Reproducible: Always

Steps to Reproduce:
1. Run a current nightly.
2. Watch the status line for "downloading n of nn messages".  Remember the count.
3. Try to find all nn messages in your mail folders.
Actual Results:  
Mail lost.  (Really it is hidden in the mbox file but unavailable because of .msf corruption and some other reason that I can not identify.  But .msf is not the only problem here.  Message filters also seem to be implicated.)

Last tested with the 20100611 SM nightly.  Problem started with 20100609 nightly.  NO problem with the 20100608 nightly.


See the following mozillazine forum posts for details (and screen captures):

http://forums.mozillazine.org/viewtopic.php?f=6&t=1919633 (for SM issues and details)

and 

http://forums.mozillazine.org/viewtopic.php?f=29&t=1919399  (for TB issues starting with the same build time frame)

Comment 1

[Wayne Mery (:wsmwk)]

seems likely a regression of Bug 387361 
http://hg.mozilla.org/comm-central/pushloghtml?startdate=2010-06-08&enddate=2010-06-09

Comment 2

[David :Bienvenu]

Are you quarantining new mail? Are you using the global inbox? And what are your filters doing? Moving messages? Copying them? Or changing flags?

Comment 3

[Kent James (:rkent)]

I downloaded a 2010-06-12 nightly, and tested simple POP3 access in new test profiles, one with quarantining emabled, and one without. I could not confirm this bug in either case. Since bug 387361 is involved with anti-virus protection, it would also be important to know what AV is in use, if any, when the issue is experienced.

Comment 4

[David :Bienvenu]

Yes, I've had quarantining turned on as well, and haven't seen any problems with pop3 mail.

Comment 5

[Daniel Kies]

After reading bug 387361, I think I have isolated the problem:  if mailnews.downloadToTempFile = true, mail is lost as I described here and in the mozillazine forums (links above).  If mailnews.downloadToTempFile = false, mail arrives as expected, as it does with the 20100608 nightly.  (I am testing with the 20100613 SM nightly.)

No global inbox is used.  Norton 360 is my antivirus application.  Message filters are set to move some new mail to specific folders on the same account.  No flags are set.  Copying of messages is not enabled.

Comment 6

[Michael Mak]

It seems to be effecting only POP3 mail. I am using the same build on my laptop but with IMAP4 mail and it works fine.

Comment 7

[Daniel Kies]

(In reply to comment #6)
> It seems to be effecting only POP3 mail. I am using the same build on my laptop
> but with IMAP4 mail and it works fine.

Yes.  I use POP3 only.

Comment 8

[Wayne Mery (:wsmwk)]

need more qa, but suspect this wants to 3.1.next

Comment 9

[[:Aureliano Buendía]]

similar to bug #571693 but with different STR

In bug #571693 I have mailnews.downloadToTempFile = true but the issue appairs only when message is move to subfolders by a filter

Comment 10

[[:Aureliano Buendía]]

(In reply to comment #9)
> similar to bug #571693 but with different STR
> 
> In bug #571693 I have mailnews.downloadToTempFile = true but the issue appairs
> only when message is move to subfolders by a filter

Sorry I mean bug #582918

Comment 11

[Wayne Mery (:wsmwk)]

new dataloss for 3.1, so patch needed ASAP. perhaps wanted for 3.1.2?  speculatively assigning to rkent.

:Aureliano, thanks for the QA analysis!

Comment 12

[Kent James (:rkent)]

Based on comment 9, I can now demonstrate on my 1.9.2 build aa simple message filter (Subject Contains filterme, Add Star) is failing for POP3 downloads when mailnews.downloadToTempFile = true  I am investigating why.

Comment 13

[Kent James (:rkent)]

Still checking my 1.9.2 build (which was very out of date), but the patch for bug 387361 was never checked into 1.9.2, so if that is the root cause I don't see how this issue affects 3.1 as suggested in comment 11. Wayne, what evidence do you have that this affects 3.1 and not just trunk?

Comment 14

[Wayne Mery (:wsmwk)]

(In reply to comment #13)
> Still checking my 1.9.2 build (which was very out of date), but the patch for
> bug 387361 was never checked into 1.9.2, so if that is the root cause I don't
> see how this issue affects 3.1 as suggested in comment 11. 

I didn't examine bug 387361 checkins, so you're probably right that this is trunk only. bug 582918 comment 0 confirms that thought. I was strictly going off of bug 582918 comment 13 and 14.

Comment 15

[Kent James (:rkent)]

I've looked at this again, and I cannot reproduce it. I'm going to remove myself from the assignee, since I have no way to make progress with this.

The bug needs a reliable method to reproduce. The changes in bug 387361 would result in subtle differences in the way that anti-virus programs interact with mailnews. For that reason, it would be important for any reporters to mention their anti-virus configuration, and to report if this bug still appears when their antivirus application is disabled.

So further progress will require additional QA effort to narrow down some reproducible conditions.

Comment 16

[(account disabled for personal attacks)]

Is anyone working on this? It's pretty annoying, I was loosing 2 to 20 mails per receive (fortunately on my test junk mail account only), so I resigned on shredder testing because of this. Happens on both machines with and without antivirus software. Tested on Win 7 32bit, 3 differently configured machines (2 desktops and a notebook). The notebook has AVG free, one of the desktops has avast Internet Security and the other is without an antivirus (it's normally not connected).

What's worse, last week it happened to my co-worker on 3.1.6. Fortunately, he was able to glimpse the sender before it got nuked, so he could ask him to resend the mail. But I wonder how many mails got lost this way without notice. So for now we banned the use of Thunderbird in our office until this gets resolved.

Comment 17

[Daniel Kies]

(In reply to comment #16)
> ... I was loosing 2 to 20 mails
> per receive (fortunately on my test junk mail account only), so I resigned on
> shredder testing because of this. ...

Have you tried setting mailnews.downloadToTempFile = false in about:config?  

This workaround has been consistently good for me since this regression started.

Comment 18

[(account disabled for personal attacks)]

I didn't try that. But it wasn't active on the 3.1.6 installation when we lost the mail. So setting it to true (and possibly the landing of the above mentioned bug) probably only makes this problem occur more frequently.

Comment 19

[Kent James (:rkent)]

I have a patch now for bug 582918, which seems to me to be the same as this bug. Does anyone have any reason to believe they are different bugs?

The issue is failure of filter moves for incoming POP3 to local folders, with mailnews.downloadToTempFile = true

Comment 20

[Wayne Mery (:wsmwk)]

I'm sufficiently convinced, having followed this for a several days.  Can be reopened if the problem continues after patch lands.

same for Bug 589092?

Comment 21

[Kent James (:rkent)]

Bug 589092 involves IMAP, so at the very least it needs to be tested separately after I complete my patch to 582918. IOW no I would not dupe it.

Comment 22

[Wayne Mery (:wsmwk)]

this bug is pop, so it's a match to bug 582918.