85632 - .msf corruption if downloading new pop3 mail while building msf

Status: VERIFIED WORKSFORME

(MailNews Core :: Backend, defect, P2)

Description

[Stefan Magdalinski]

From Bugzilla Helper:
User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)
BuildID:    20010607

if a .msf is being rebuilt, and mail starts downloading at the same time, you 
get severe corruption including:

1. mismatches between mail index pane , and message contents (i.e. wrong sender 
and subject line for the body of the message).
1a. sometimes total misalignment of message bodies. 
2. mail filter rules break, and everything gets dumped in Inbox.
3. duplicate copies of some messages. 

I'd like to suggest the dataloss keyword for this: I'm not certain, but I think 
I have lost some messages. certainly the breaking of the mail filter rules, and 
dumping 300 messages in my Inbox is a dataloss.

Also, if you've got a complex mailbox, (30 folders, 25 filters, 4 years worth 
of messages), deleting all your .msfs is more than a royal pain. I think this 
is important to fix before moz is a real-person usable product.

Reproducible: Always
Steps to Reproduce:
1.delete .msf file for Inbox (although I notice normally during the periodic 
rebuilds)
2. start mozilla-mail
3. while the summary file rebuilds, start downloading new messages

Actual Results:  .msf corruption.

[e-mail] discusses what appears to be the same issue in #50155 
(but I checked this isn't a dupe).

Comment 1

[Oliver Klee]

Adding dataloss keyword.

Comment 2

[David :Bienvenu]

it's the responsibility of the back end to not put this bogus data into the
database - it's not a database problem. The way this is supposed to work is that
if we detect that we have to reparse the inbox before downloading new mail, we
should postpone the downloading of the new mail until we've finished parsing the
folder. Cc'ing Navin in case he wants to look at this while I'm on vacation. My
suggestion is that we hook into the url listener in nslocalmailfolder to get new
mail in the case that we have to reparse the inbox.

Comment 3

[Navin Gupta]

reassigning to self. nominating. 

Comment 4

[Sheela Ravindran]

changing bug status based on navin comments below 

Comment 5

[Navin Gupta]

Attachment: proposed fix.

Comment 6

[Navin Gupta]

I have added a boolean flag to nsIMsgLocalMailFolder so that we know if 
we can perform biff from front-end. This is set to true if we have to 
parse the inbox on start-up, in that case we call getmessages from the 
back-end. looking for review from david. 

Comment 7

[Navin Gupta]

Attachment: proposed fix, v2

Comment 8

[Navin Gupta]

Also made it so that if the user clicks GetMsg() and parsing is going on, then 
the GetMsg() is delayed until the parsing is over. 

Comment 9

[scottputterman]

Adding nsbranch but it looks like we should check this into the trunk for a day
or two, try it out and then try the branch, once this gets reviews.

Comment 10

[Navin Gupta]

I need review, david. 

Comment 11

[David :Bienvenu]

it looks to me like this patch will make us do a get new mail on any INBOX
folder that we have to reparse, for whatever reason we had to reparse (e.g.,
just clicking on the inbox folder) - that seems wrong. Am I misreading this?
Basically, you set that mCheckForNewMessagesAfterParsing flag whenever we
reparse the inbox, which is wrong - we could be just reparsing because the user
clicked on the inbox.

Comment 12

[Navin Gupta]

point taken, i will attach a new patch. However i was thinking if parsing of
inbox in a session usually will be done once and since we always select 
inbox on startup it should be fine (would be doing GetMsg()), need not be 
true always. 

Comment 13

[David :Bienvenu]

only if it's your default account and we select it on startup - that's not true
for me, for example.

Comment 14

[Navin Gupta]

Attachment: proposed fix, v2

Comment 15

[Navin Gupta]

ok added another flag to keep track of whether GetMsg() is triggered.

Comment 16

[David :Bienvenu]

+  if (mParsingInbox)  //make sure we turn off this flag even if parsing fails
or else it will be a deadlock
+    mParsingInbox = PR_FALSE;

can just be 

mParsingInbox = PR_FALSE, right?

Comment 17

[Navin Gupta]

I mean it will always be false for other folders, well trivial anyway so I will
change it. 

Comment 18

[David :Bienvenu]

one last thing (sorry)
+    NS_WITH_SERVICE(nsIMsgMailSession, mailSession,
+                       kMsgMailSessionCID, &rv);

should use do_GetService, and if there's a contract id, you should use that too.

do that, and r=bienvenu

Comment 19

[Scott MacGregor]

sr=mscott modulo david's comments.

Comment 20

[Navin Gupta]

fix checked in. 

Comment 21

[David :Bienvenu]

you should leave this open, since you need to check it into the branch too, right?

Comment 22

[Navin Gupta]

checked into trunk only. 

Comment 23

[Stefan Magdalinski]

just to confirm that this works for me on 

Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:0.9.1+) Gecko/20010628
(I trashed my Inbox.msf, and downloaded a mail). 



and to say:

 thanks guys. I've been using mozilla as my main mail client since M16 (I've
been a netscape user since the days I needed to be able to access the same
mboxes in pine and netscape. I'm not a proper tech any more; watching the
progress of this has been a revelation that the whole project works beautifully.
I only wish I'd reported it earlier, when I first spotted it. next time... 

again, excellent work. 

stef

Comment 24

[Navin Gupta]

*** Bug 73039 has been marked as a duplicate of this bug. ***

Comment 25

[lchiang]

adding vtrunk keyword to indicate the bug has been fixed on the trunk.  Bug left
open for tracking checkin to branch (nsbranch) when appropriate.

Once bug has been fixed on the branch also, pls remove vtrunk keyword.

Comment 26

[esther]

I'm trying to reproduce this with builds on 6-25, but I can't.  In order to
verify the fix, I need to see the problem.  Is there anything not mentioned in
the original scenario that I need to do to see this problem.  
I removed my .msf file for my POP inbox
I have (2) filters to filter the sender.
I launch profile again.  With another system send my pop account msgs that will
be filtered when I do Get Msg.
I login to the POP account.
The summary file is building, I click Get Msg several times while the summary
file is building.  

Everything comes in fine, no mixups with content, newly sent msgs that meet the
filter criteria get filtered, previously downloaded msg that meet criteria
didn't filter.  I have 395 msgs in inbox and (3) folders, 2 filters set up.
If I need the much larger account and 30+ folders to see this please let me
know.  Navin, if you have a test account that's setup to see the problem please
let me know.  Still waiting to verify.

Comment 27

[esther]

Stefan, you confirmed this is working for you with a trunk build after 6-28,
Navin checked in the fix late on 6-28 and I'm not sure if the build you took had
the changes.  If it didn't have the changes, then I'm not sure why it's working
for you with that build, we could be missing something.

Comment 28

[Navin Gupta]

esther, stefan has already verified the fix using the nightly builds. 

Comment 29

[Navin Gupta]

Attachment: patch for the case when user clicks on inbox and biff gets fired simultaneously.

Comment 30

[Navin Gupta]

David, this last patch should go in, right ?

Comment 31

[David :Bienvenu]

yes, though you might want to check for null on local inbox.

Comment 32

[Stefan Magdalinski]

Esther, I put the build string in the comment when I verified, so you should be
able to tell if the fix was in.

I was watching the tinderbox very carefully to see when it went in, so I think
it was ok. I definitely haven't had the corruption since then.

as for reproducing the original bug: my Inbox has 2500 messages, and I have many
folders. If it's possible that it makes a difference, I also download from 2
different POP accounts. 

stef

Comment 33

[Matthias Versen [:Matti]]

*** Bug 89942 has been marked as a duplicate of this bug. ***

Comment 34

[Aleksey Nogin]

Copying from bug 89942 where I described a slightly different scenario (to give
another test case for this fix):

1) Create a big POP Inbox that will take a while to compact (put it on a slow
NFS server if necessary :-) ).
2) make sure you have new mail, but do not do "Get messages".
3) Compact folders
4) While Inbox is being compacted, do "Get messages".

Expected: new mail normally arrives and is copied during compacting, or new mail
retrieval is delayed until compaction is finished, or "get messages" is disabled
during compaction.

Actual: new mail is lost.

I have "leave messages on server" enabled, so I could see the message that was
lost using a different MUA, but Mozilla never showed it to me, so I would
imagine that without "leave on server" this would be a dataloss.

My guess is that this may also happen with movemail, but I haven't tried it.

I was using Mozilla 0.9.2 (from mozilla.org RH7 RPMs) on RedHat 7.1

Comment 35

[Navin Gupta]

The compact part of this bug is dealt with in another bug. 

Comment 36

[Sheela Ravindran]

Marking this bug as verified based on reporter comments on 2001-07-04 07:26.
Esther spend time on trying to reproduce this problem and was not able to.  
Since the reporter has commented that it has worked for him and also based no 
further reports in the bug regarding the initial bug. If this problem occurs 
again please reopen.  
But any other scenarios and compact problems log a new bug or there are number 
of bugs already reported which you can see if you run a query for compact bugs. 
 

Comment 37

[Stefan Magdalinski]

oh dear. I've recently (I would guess sometime after early October) started 
seeing this again. happened this morning:

started downloading mail, Inbox.msf starts rebuilding and I end up with all my 
new mail dumped into my inbox, instead of filtered into the right folders. 

I'm not sure if it's the exact same problem, but the effect is the same. build:

Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:0.9.5+) Gecko/20011015

Comment 38

[Christopher Hoess (gone)]

milestone->0.9.7

Comment 39

[Navin Gupta]

>started downloading mail, Inbox.msf starts rebuilding and I end up with all my 
>new mail dumped into my inbox, instead of filtered into the right folders. 

>I'm not sure if it's the exact same problem, but the effect is the same. build:

This bug was about downloading pop3 msgs colliding with building summary (.msf 
file) and I checked just now, it works. Filtering to right folders is a separate 
issue, log a new bug - it could be that it was unable to write to filter 
destination (folder) for some reason. 
 

Comment 40

[esther]

This bug is morphing into other bugs.  The original bug as stated,  is OK on a
6.2 rtm and 11-19 nightly build.  Stephan can you write a new bug for the
filtering problem.  Can you give steps too. I tried this with my POP account to
see if the original problem and the filtering problem could be reproduced with
no luck.  I had leave messages on server checked. If I remove my .msf file for
the Inbox, old messages that match the filter criteria but were not filtered
before are not filtered during the rebuild of the .msf file, but I didn't expect
them to be.  Messages that had been filtered before did not show up in my inbox
during the .msf rebuild.  Also note, with the 6.2 rtm build I persistantly
clicked on Get Msg while the .msf file was rebuilding, but it would not bring up
the password dialog until it was done building the .msf file and no messages
were misaligned.  With the 11/19 nightly build, the .msf file build so fast I
didn't have time to click the Get Msg before it was finished.  So for now, we'll
resolve this again as worksforme and ask if you could let us know if this (msg
content getting misaligned with selection in thread pane) has happened with a
more recent builds and if the .msf file is being rebuilt because you remove it
or is it happening on it's own. 

Comment 41

[Stefan Magdalinski]

ok. 

I took on board navin's comments, went to file another bug, and then found 
http://bugzilla.mozilla.org/show_bug.cgi?id=50155

which describes the problem pretty closely. I think what we're seeing is a
different cause producing a nearly identical effect.

I think we should leave this bug resolved, and keep 50155 open. I don't yet have
steps to reproduce, but I'm trying... see my comments over there.

I'm seeing this with 18/11 nightly.

Comment 42

[esther]

I read bug 50155 and agree this can be verified as worksforme.  Thanks for the
followup.  Verified.