129935 - overwrites subsequently viewed mails with another viewed mail

Status: VERIFIED DUPLICATE of bug 122361

(MailNews Core :: Backend, defect)

Description

[ulf schreiber]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.8) Gecko/20020204
BuildID:    2002020406

Sometimes (i know "sometimes" is a very bad word in bug reports) i read one mail
in a mailinglist folder and after that all other mails i try to read in that
folder get overwritten with the header and contents of the first one.

Even the sender/subject/etc fields in the sender/subject/etc pane change to the
values of that first mail. Only thing that is left from the mail that should be
there is its length: The contents of that first mail that overwrites the others
is clipped to the new length, or is filled with header data and body of the next
mail (or the same, i'm not sure because after some iterations of this overwrite
stuff they all look basically the same).

I tried rebuilding the msf but that doesn't help, because the corruption goes
down to the folder file (the one with header\nbody\nheader\nbody...). Why is
there anything happening except addition and deletion ?

Reproducible: i have no clue how to reproduce it so far, happened more than one time

Comment 1

[David :Bienvenu]

this isn't a mail database problem (the mail database is the .msf file, which
stores the headers for each message, generated by parsing the mail folder. If
regenerating the .msf file doesn't work, then the mailbox store itself is
corrupted). 

I assume you're using pop3 mail, and that these are local folders, and not imap
folders. If that's the case, the local folder separates its messages with a line
starting with "From " - it sounds like somehow all those "From " lines have been
overwritten?

I have no idea how this could happen. Are you using mail filters to move
messages into these mailing list folders? Do you compact the local folders, or
have auto-compaction turned on? Do you ever read the messages in the mailinglist
folders as messages are filtered into them? The only way to figure out what's
going on here is to be able to recreate this problem, so we're definitely going
to need help here.

Comment 2

[ulf schreiber]

Your assumptions are right, pop3, local uncompactet folders used as 
target for filters, and reading while new messages are sorted 
into the folder is quite likely to happen.

"From"-lines are there (the mail count after a reparse was the same as 
before, iirc), but i have yet to find out wether their offsets have 
anything to do with the original ones. Also i get "From"-lines without 
a previous newline: There are some cases where somewhere in one of the
"rogue" copies of the one mail that overwrites others suddenly a new
copy begins, with its own "From"-line and header, but no newlines 
preceeding the "From".

Example:
<-------body-of-first-copy----------||-----begin-of-next-copy------>
Neither the random thing nor the XOReFrom - Sun Mar 10 04:07:13 2002
X-UIDL: (@^"!<(<!!Lco"!VNg"!
X-Mozilla-Status: 0011
X-Mozilla-Status2: 00000000

Comment 3

[David :Bienvenu]

right, the problem is that the "From " doesn't occur at the beginning of the
line. The problem is figuring out what caused this. But first of all, you should
upgrade to a more recent mozilla build just to make sure that this hasn't been
fixed - I think Navin has made some fixes in this area. Then, we need to try to
figure out how to reproduce this. My strong suspicion is that this has to do
with reading unread messages in the target folder of a filter while messages are
getting filtered into that folder, so you could either try doing that and see if
the corruption occurs immediately (or after a compaction), or if NOT doing that
results in not having any more corruption.

Comment 4

[ulf schreiber]

OK, i still can't reproduce this at will, but i got another case of this mail 
overwriting. This time with build ID 2002031103. It also started when there was 
mail filtered into folders, but this time it occurred in multiple folders and 
The mails were overwritten with a mail that was going to a folder that i did 
not enter at all today. Some tests with the error showed that the mails are not 
overwritten on read: when i restarted mozilla there were some still unread 
mails in the summary view that showed to have the corrupt content ([ot:]This 
time it's a fragment from a Cornelia Sollfrank interview). So i went on to 
check a folder file, and even there the headers/contents of new unread mails i 
could see in the summary view where missing and instead i found some beginnings 
of the interview of different length.

It seems to me that the header information is correctly added to the msf file, 
but the write out to the folder file gets the wrong pointer, but correct length 
(i assume that there are "begin of mail" - offsets in the .msf)

PS: I also checked the mail server (d/led the mails with MS-OE), and they are 
correct.

Comment 5

[David :Bienvenu]

Ulf, which folder or folders were the messages overwritten in? Was it the inbox?
You're right that we store the offset into the folder of the start of the msg in
the database.  When we filter new mail, we write the message to the inbox and
then notice that we have to filter it - at that point, we copy the contents of
the message from the inbox to the filter target folder,  and truncate the inbox
back to the start of the filtered message. If that truncation were to fail, we
could get confused about offsets into the inbox of the next message.

Comment 6

[ulf schreiber]

Congratulations! You were right about the failing truncation of the inbox file! 
I never recreated the inbox.msf after some of this overwriting buisness, 
because according to the summary information, the mails got correctly filtered 
to the folders, but lost their content on the way. Now that i recreated the 
inbox.msf all those mails appeared in the inbox. 

Hehe, seems that this bug does no longer qualify as "critical", since no data 
is lost anymore.



Some things (maybe of interest) i noted down before i realized that the mails 
were hidden in inbox:

So far both the mails that got overwritten as also those whose header/content 
was used to overwrite were mails that got filtered to mailinglist folders 
(though inbox traffic is not so high, it could just be chance).

At what point are the summary files or the displayed data in the summary pane 
(what's it called, the "sender"-"subject"-"date" pane?) written in the 
filtering process?  They do contain the correct data as long as i do not 
recreate the summary files.



PS: is it true that the original 4.x communicator used the same approach? Now 
as i see somewhat clear what happens in Mozilla i remember old filtered out 
mails flooding my Communicator inbox in large quantities from time to time, 
appearently after an automatic summary reconstruction.

Comment 7

[David :Bienvenu]

I believe this is a dup of a bug Navin has fixed.

Comment 8

[Navin Gupta]

Now the truncation of Inbox should never be a problem because we are sharing
inboxStream. 

*** This bug has been marked as a duplicate of 122361 ***

Comment 9

[[e-mail] (gone - use [e-mail] instead)]

verified dup.  [e-mail], please re-open bug 122361 if you see that
exact same problem, still.