Closed Bug 218433 Opened 21 years ago Closed 21 years ago

Inbox lost regulary. Suspect problem with bayes filters and compacting?

Categories

(MailNews Core :: Database, defect)

Product:
MailNews Core
Component:
Database
Platform:
x86
Windows 2000
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: bugzilla, Assigned: Bienvenu)

References

Details

(Keywords: dataloss)

Attachments

(4 files)

My prefs file
22.38 KB, text/plain
Details
screenshot of some funky mail goings-on
94.00 KB, image/gif
Details
Filter rules for the evonzee@ account.
1.29 KB, text/plain
Details
proposed fix
2.68 KB, patch
mscott
: superreview+
Details | Diff | Splinter Review 
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5b) Gecko/20030827
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5b) Gecko/20030827

I have 6 pop3 accounts that I check with Moz mail.  All have the bayes filters
enabled.  Frequently, when I start Moz, it downloads mail from each of the
accounts, but stops filtering the Inbox halfway through.  When that happens, I
lose all contents of the Inbox of the primary account.

Things I've noticed about the problem are:
I think it usually happens on the next restart after Moz asks me to compact
mailboxes.  At least, that's what happened this time.
It has nothing to do with the content or size of the incoming mail.
I thought it had to do with reading a message while more were downloading, but
it has happened without me reading anything.

Other information that might be relevant:
I have ~20 regular non-bayes filters that file bugmail, etc.
Message body is viewed as Original HTML.
When the crash happens, the messages still are listed in the box, but none are
openable.  Some contain junk data, others are completely blank. 
If I delete the Inbox.msf, the messages that were most recently downloaded
become viewable, but the old messages are gone.

Reproducible: Sometimes

Steps to Reproduce:
1. Have all mozilla windows closed, including quick load (which I don't use)
2. Open mail via the "mozilla.exe -mail" shortcut

Actual Results:  
Moz checked the mail on each account and downloaded new messages.
Bayes filters executed as each account completed download
All email in Inbox, including very important email from Verisign, disappeared. :(

Expected Results:  
Mailbox should not have been corrupted.  All mail should have remained intact
and I should not have sprouted a new patch of grey hair.

Other bugs that might have some bearing on this bug are:
bug 159876
bug 188054
bug 193916
bug 201715
.. but none of these seemed to have the same exact problem as I have.
you don't have these pop accounts using the same local mail directory, do you?
To do that, you would have had to change the local mail directory for the pop3
accounts.

If not, I doubt that having multiple accounts is an issue. The auto-compact code
used to cause problems like this, but I thought that had been fixed with folder
locking. I have a similar configuration, and even when I allow the auto-compact
to happen, I don't have this problem. Very few messages ever end up in my inbox,
however, since personal mail is moved to another folder, and non-personal e-mail
tends to be junk, and moved to the junk folder by the spam filter.
Status: UNCONFIRMED → NEW
Ever confirmed: true
I tried this - I get an alert that the folder is busy (the inbox) and that I
should wait until the operation (the compact) is finished. That's on the get new
mail part...it is the primary, first account that's having the problem, right?
Do you ever see this or a similar alert? I also sometimes get alerts that the
mail filter can't move the message because the destination folder is being
compacted.
No, each of the accounts has its own folder structure.  I created each account
using Mozilla, and didn't change anything related in prefs.js.  Upon inspection
of prefs.js, each mail.server.server[x].directory is different.  This is what
you were talking about, correct?

I do occasionally see the messages about busy folders, but I usually don't.  No
such alert appeared this time.  And yes, it is only the primary account that has
the problems, as far as I can tell.  That account is responsible for 90% of the
traffic I get, though.

More things that might be relevant:
I do have all accounts set to automatically check for new mail on startup.  
All spam is moved to a single folder on Local Folders, rather than each account
having its own Junk box.

I thought that perhaps, since Moz is checking for new mail on startup, that it
would start connecting to the pop server, and only after the connection was
established, it would offer to compact?  Maybe then the locking system would be
bypassed, since the mail download had already started?  I'm just grasping at
straws here, since I have no familiarity with the details of the locking system.
yes, that's what I was talking about with local directories.

Even if you automatically download new msgs, the auto-compact prompt still
happens before we download pop3 messages, in my experience.

The filtering stopping is a clue, I guess, though I don't know why it would
happen. When you say you lose all the contents of your inbox, have you looked at
the actual INBOX file to see if the messages are still there, but it's just the
INBOX.msf file that's lost the messages? Moving aside the .msf file would
restore the messages in that case. Also, compacting will clear out the message
list, and then rebuild it when the compact is finished, making it appear
temporarily that the messages are gone. I don't think that's what's happening
here, but I thought I'd mention it. Also, do you have a folder called something
like nstmp now? That will get left around if a compact is aborted mid-way...
When the inbox disappears from within Moz, the actual Inbox file is emptied of
all content except for that which was downloaded during the download session
where the problem appeared.  So, if I have 20 messages in Inbox, and while
downloading 5 new ones, the inbox eater strikes, all that the Inbox file will
contain is those 5 new messages.  The msf file still references the old
messages, but they're gone from anywhere else.  Deleting (or moving) the msf
file results in being able to read the 5 new messages, but the old 20 are now
completely gone.

There are no files or folders in my profile directory called nstmp.

I suppose I don't really *know* whether the filtering was complete.  I am
assuming that to be the case because 2 of the new 5 messages are tagged as spam
but were not moved into the junk folder.  Two more of the 5 should have been
spam, but I can't tell if they weren't tagged because of insufficient
training.dat or because of filter crash.  Maybe the filtering was complete, and
Moz was busily copying the mail into the junk folder, except it was locked by
another account's filtering?  As I said, all 6 accounts pool their junk mail
into the single Junk folder under Local Folders.

As I think about this bug, I am becoming more certain that there is something
wrong with a setting in my prefs file (or something in my profile) somewhere.  I
say this, because I recently moved to a new machine and installed a fresh
Mozilla, and then copied my profile from the old machine to the new one.  The
new machine has the same problem.  I had been experiencing this problem since
Moz 1.2 on the old machine, but none of my co-workers have ever had this happen.
 And, I'm sure that there'd be a direct dupe of this bug if other Moz users were
experiencing it.  Some combination of preferences must be at fault here.  Would
it be of any help for me to attach my prefs.js file?  I would bet that I'm using
some options that most people don't use, like full headers display, pie menus, etc.

Oh!  As I typed the previous paragraph, I also realized something more
significant that sets me apart from the other Moz users here - I am using
encrypted POP3 on port 995, while everyone else is using standard pop3 on 110. 
Could that have something to do with it?
Keywords: dataloss
Attached file My prefs file — 
Rather than wait for your response, I thought I'd just attach the prefs file
and if it comes in handy, it will save us one exchange of bugmail.  Usernames,
emails addresses, and salts mentioned in the prefs file have been modified from
their original values to preserve my security, everything else is exactly the
same.
Do you have a virus checker installed on either machine? I always forget that
some virus checkers have been known to delete the INBOX :-(
Yes, both machines run Symantec Antivirus, Corporate edition.  All machines on
our network run the same software.  Is this one of the Inbox eating AV packages?

I just had my inbox eaten again, I think.  I've started moving everything out of
there manually, as soon as it comes in, to avoid this problem.. but there was
one message I don't think I had moved yet, which is now gone.  I have the Inbox
and Inbox.msf files zipped from 30 seconds after the breakage.  Looking at the
Inbox file with UltraEdit, I notice that line 7147 has 165005 bytes of NULL
characters (hex 0x00), and then the standard From - (date) delimiter.  Maybe
this is part of the problem?  

The zip file is too large to be attached to Bugzilla, so you can download it
from http://base10.org/devel/Inbox-postcrash1.zip
I can't remember which Anti-virus programs caused this problem, sorry. Older
ones in particular seemed to cause it. Also, with some AV programs, you can tell
them to ignore files with certain extensions, like no extension, as in the case
of mail folders like INBOX. You might try that.

You should never get 0x00's written into your mail folder - Mozilla won't do
that because we treat the data from the server as null-terminated strings.
I have added the INBOX files to the list of excluded files in SAV.  However, we
are using the most recent version of the product (as far as I know) and it seems
to me that the likelihood of the AV causing this problem is minimal (not least
because all of the other users here that have no problems with Moz Mail are also
using SAV.)

The nulls are an interesting find, then.  There are hundreds of thousands of
them in my inbox file.  Is there anything I can do to determine the origin, or
has this bug gone as far as I can push it? 

I will definitely add more comments to this bug if the inbox gets eaten again,
now that SAV isn't scanning it.  
Hi David,

Unfortunately, I return with bad news.  My inbox is gone again.  I saved the
inbox data again, but I'm not going to post it unless you need it.

The file contains more null (0x00) characters, but not as many as the last time.
 There were only about 25,000 in this one.  They again appear directly before a
"From - (date)" line late in the file.  Also, the bodies of some of the messages
were jumbled around.  The Inbox file's first line was halfway through one of the
new messages received in the download session where the mailbox corrupted. 
There was no From - line before it.

I had been keeping one test message in the Inbox in the hopes that the content
might be findable after a dataloss event.  It contained the string
"xoepxoepxoepxoepxoep".  That string does not appear in any file currently in my
profile, meaning that the data is completely gone.

Any more thoughts?

Thanks, by the way, for your help in trying to diagnose this bug.
Did an auto compact happen in this latest case? I assume your file system is the
standard windows file system on your local machine and you're not low on disk
space or anything like that? Is the account in question the Eric+ tritechnet.com
one? 

What are these ext.checky prefs about? e.g., ext.checky.pref.global.tempdirectory"
No autocompacting happened in the latest case.  Last night, however, I was
prompted for an autocompact, and I let it run.  This was the first time I'd
opened Moz mail since that compact.

My filesystem is NTFS under Windows 2000.  The disk is an 80GB IDE drive, and
has 65GB free space.

The account that loses its inboxes is indeed eric at tritechnet.com.
(mail.server.server3, mail.identity.identity2)

I assume the ext.checky prefs are added by the Checky plugin from
http://checky.mozdev.org/.  I am a web developer, and I use the plugin to easily
validate local intranet pages against multiple validators before I go live with
them.
I'm clueless. Can you try turning off auto compact completely for a day or two
so we can rule that out completely? If that doesn't help, I'd also suggest
turning off the virus checker completely but I can understand if you're not
comfortable with that. I'll look at the bayesian filter code and see if moving
junk to a different account could possibly cause problems.
Just thought I would report back.. I've had autocompact off for the last two
days and have had no problems whatsoever.  Think I should turn it back on and
see if it creates a problem?  That would at least suggest some correlation
between AC and the inbox loss issue.

I asked around and nobody else here uses the autocompact feature.  In fact, they
were unaware that compacting was required at all. (I wonder if that's true for
most mozmail users?)  So I'm the only one here that has ever compacted a mailbox. 

As far as bayes filters go, they're still working fine, so perhaps they should
be eliminated from consideration as the cause.
If you think that's a long enough test, sure, turn auto-compact back on.
However, we can't remove the bayesian filters as a contributing cause unless you
turn them off and the problem still occurs. My guess is that it's the
combination of the two. You could try to turn off the automatic moving of junk
mail to the junk folder and just do that occassionally by hand or use the Tools
| Delete mail marked as junk command to delete the junk and see if that helps.

The other thing that would be interesting to see if if new mail is downloaded
*before* the auto-compact prompt pops up. In my profile, new mail is only
downloaded *after* the auto-compact prompt.
Hi David,

I'm glad I didn't get around to re-enabling compacting - my Inbox went kablooey
again today, and I haven't compacted for over a week.

Unlike the previous experiences, the old messages are OK, it's the new message
that are borked.  Examining the contents of the Inbox file, I see that the new
messages are present, but Moz interface refuses to display the content.  

As has been the case with my prior corruptions, the Inbox file begins halfway
through the content of a message (which happens to be spam) and lacks the "From
- " stuff.  There is also ~30k of 0x00 near the end of the file (again...).

This time around, I had opened mail while I was actively browsing in another
window.  While the mail checked, I opened two new browser tabs, and then
switched focus back to the mailnews window, which had stopped filtering spam,
and erased some of the new messages.  However only *some* of those new messages
were borked, there were a few that came through OK.

Here's a *VERY* interesting tidbit that I thought of doing while typing this
bug.  The position in the file where the nulls appeared, is where the half-mail
that was at the top of the file should have appeared.   I used UltraEdit to cut
the first 30k of the file (the same number of bytes as there were nulls) and
paste it over the nulls, and saved... and BOOM my mailbox was OK!  However, all
messages that were "deleted" were now duplicated.  It was particulary
interesting that once those first 30k bytes were cut, the first line of the
mailbox was now a "From - (date)" line again.

So, in summary, it looks like compacting has nothing to do with it.  Somehow
parts of the file are getting moved from their intended position and prepended
on the mailbox file by some other process, and their position in the mailbox is
replaced with nulls.

Hope this leads to some new thoughts on the problem.
>However, all messages that were "deleted" were now duplicated. 

Scratch that.  *all* messages, new or old, were duplicated.  One copy seems to
be fine, and the other has the MIME info and the headers displayed in the body.
  Killing the Inbox.msf and restarting Mail eliminated all the dupes and
malformed messages.

OK, thx, it's good to know that auto-compact is not involved. I'm still clueless
as to what could be writing nulls into the beginning of the mail folder. One
thing I wonder is if you're getting spam with null bytes in it. Some pop3
servers won't give you nulls in a mail message (technically, it's illegal), but
some will.

Let me be sure I understand you here:

You say - "While the mail checked, I opened two new browser tabs, and then
switched focus back to the mailnews window, which had stopped filtering spam,
and erased some of the new messages."

So, the whole get new mail process had finished, including the moving away of
the junk messages, and you went and deleted some of the newly arrived messages?
At the point that you deleted/erased some of the messages, nothing else should
have been going on, at least in Mozilla...
My pop3 server is Qpopper (version 4.0.3).  I do not know whether this
particular server allows null characters in message content.

>>"switched focus back to the mailnews window, which had stopped filtering spam,
and erased some of the new messages"
>"So, the whole get new mail process had finished, including the moving away of
the junk messages, and you went and deleted some of the newly arrived messages?
At the point that you deleted/erased some of the messages, nothing else should
have been going on, at least in Mozilla..."

No, I apologize for my lack of clarity.  What I meant to say, was that Moz had
had begun the spam filter process, had stopped halfway through that process, and
had "erased" the bodies of some of the messages.  When I say, "erased," I refer
to the symptom that I have been experiencing all along - message bodies are
gone, but the headers remain until I remove the .msf, at which point they go
too.  I took no action with regards to erasing any mail.  As soon as I
alt-tabbed back to Mail, I noticed that some messages were marked Junk but not
moved away (red flag!) and that I was unable to read the bodies of some of the
new messages.  I then closed Mail and did the research I outline in the "*VERY*
interesting" paragraph of comment 17. (detailed again below)

The nulls are not appearing at the top of the mail file; rather, they're
appearing at some later point in the file.  The data that *used* to occupy the
same space as the nulls is transported to the head of the mail file. 
(ASCII)graphically, this might be represented as the following, where the
repeated numbers each represent (for example) 1k blocks of an email (defined as
the content from a "From - " line, to the line before the next "From - " entry
(or EOF), inclusive)

Original:
1111111111112222222222222233333333344444444444555555555555555666666666666

Post Corruption:
555661111111111112222222222222233333333344444444444555555555555NNNNN6666666666

NNNNN represents 5 1k blocks of NULL that take the place of the three blocks of
message 5 and two blocks of message 6 that get moved to the beginning of the
mailbox file.

Hope that wasn't too confusing.  With all the numbers I just typed, I'm getting
confused myself.
Eric, would it be possible to turn on pop3 "leave msgs on server"? If you leave
messages on server (if only temporarily), when we might be more easily able to
recreate this bug, and see if it's related to receiving particular messages. You
can configure Mozilla to delete messages from the server when they're deleted
from the client, and/or you can configure Mozilla to delete messages more than
XX days old (in 1.6). It occurred to me that perhaps getting messages with nulls
in them might cause us to do unpredictable things.

About the Nulls and data getting written to the start of the folder, it's as if
some code is moving the file pos pointer to the beginning of the folder and the
code that's writing to the folder isn't correctly positioning the pos pointer to
the end of the file. I can't think what would do that...compaction is always a
suspect, because it causes the whole mailbox to be rewritten, but it sounds like
 we've partially ruled it out.



Hi David,

I have enabled the "leave messages on server" and "Delete messages when deleted
locally" options on my primary account.  I'll let you know when (if?) it breaks
again, whether there were nulls in the mailbox file on the mail server (I do
have root access to the machine.)

As far as ruling out the compact process, I'd agree that it's highly unlikely to
be the cause of the problem, because I had the auto compact disabled and hadn't
done a compact the last time I had the problem.
If it does break again, we can see if it's reproducible by shutting down,
cleaning  up, deleting popstate.dat, and restarting - that will cause us to
redownload all the messages on the server again. If there's something about the
messages that causes the bug, that will help reproduce the problem.
Status: NEW → ASSIGNED
Hi David,

Just thought I'd check in.  I haven't seen any inbox corruption since my last
post.  I am currently using build 20030916 (is this 1.5 RC1?  I can't remember.)
 I plan on moving to 1.5 as soon as it is released.  I suppose if I don't have
any problems for a month we can close this bug as WFM and reopen it if the
problem ever appears again.
Hi David,

I spoke too soon.  My inbox went strange again.  The inbox file contained 75k
null characters, and this time the break happened to be on message borders (so
it wasn't immediately noticable)

I had the "leave mail on server" option checked, so I created a new profile,
configured my pop account, and checked. No corruption, nulls, or anything like
it.  Unfortunately, I hadn't checked the "leave mail on server" option on the
new account, so I wasn't able to verify the contents of the mail spool at the
server.

However, since the same mail download didn't create the problem on the second
account, can we rule out message content as a cause?
Bummer. Do you still have auto-compact turned off? Grasping at straws, here, do
you have win2k's file system set up in any special way other that fat32? ntfs?
compaction or encryption?
Yes, autocompact is still off.  I have not compacted at all since my last report
(9/23).

Local filesystem is NTFS, no compression, no encryption.

This really *is* an elusive bug, isn't it. :P
elusive to put it mildly. Heck, I'm starting to wonder if it does have to do
with using SSL on port 995 - in theory, that should have no effect but maybe
there's a bug in there somewhere.
OK, I'm going to go back to regular pop3 on 110.  I'll be sure to update the bug
if the Inbox dies again.

My pop server isn't exactly the most reliable in the world.  What do you think
would happen if an SSL pop connection was open, and the server fell asleep
halfway through, and terminated the connection?  Might the message data get
corrupted but written to the mailbox file anyway?  Not that I think that has a
high likelihoodof being the source of the problem, but at this point, any idea
seems like a good one. :P
if the server is flakey, and drops the connection, it can definitely cause
problems, from what I've heard. I don't see how it could cause the inbox to get
lost, however.
Hi David,

Well, the inbox went south again.  This time, even after my null copy/paste fix,
I still lost a few messages.

When the problem occurs, the inbox isn't totally lost.. the message titles still
show, and a few of the bodies are still readable, but most are not.  View-source
on messages usually shows an empty screen, and selecting messages on the list
shows headers that were actually for different messages on the same list.  I'll
attach a screenshot that hopefully will show you what I mean.

I doubt that the server is the problem, since everyone else here uses the same
server with no problems, but due to lack of possibilities, I though I'd throw it
out there.  Besides, even if the server dies halfway through a download, it
would be a bad thing to lose a user's entire inbox (because most people I know
leave most of their mail there!)  I don't know of a way to check whether the
server was the cause of the problem anyway.

    
    

    
  
Hi David,

Just to keep you posted, upgrading to 1.5 didn't change anything.  This morning
I had another mailbox incident.

Lately, the cutoffs on the NULLs have been on message boundaries, so just
deleting the .msf can fix them (assuming I remove the nulls from the file), but
I think that's just chance.  So I always move enough data from the front of the
file to overwrite all of the nulls, and I seem to be coming out OK.  I believe
that even after this fix I still lose messages (the message I was keeping in the
Inbox to test whether mail disappears was gone) but I can't swear to it - I
might have accidentally deleted that placeholder message myself.

My upgrade to 1.5 routine was pretty standard:
Remove 1.5b via add/remove programs
Go to program files\mozilla.org\mozilla and remove everything except the plugins
directory
Install new version
Install my standard XPIs (which consist of calendar, piemenus, checky, JSConsole)
Set moz as default mail and web client
Back to business.

    
    

    
  
Hi David,

I have additional new information for you.

The corruption does not always involve null characters.  In this most recent
breakage (1 minute ago), a newly arrived email overwrote an old email that was
in the Inbox, but had been moved away by a filter.  Since the new email was
shorter than the old one, reading the mailbox file makes it appear that the
ending of the old email is tacked onto the tail of the new message.  The new
message therefore did not appear in the Inbox list, since (I assume) the .msf
file knows that the message at position 1 is deleted.

I also have no way (that I can think of) to recover from this problem, since I
don't know where in the file to place the new message, and the headers and first
part of the body of the overwritten message are completely gone.  I know this
because it was a system status message that I receive from one of our servers on
a daily basis, so I knew some text that would have appeared in the headers.  I
was unable to find that text in the Inbox file, though.

So, the exiting summary is that the problem might be worse than I thought it
was.  Nulls aren't necessary for the problems to occur, either - unless this
latest problem is a separate issue from the one I started out having.

    
    

    
  
Another thought I had with regards to work flow:

I am a compulsive window closer (or so my coworkers say.)  If I'm not using an
app Right Freaking Now, it's gonna get closed.  There are *never* more than 4
items open on the task bar, and two of them are usually filesystem browsers. 
Therefore, I would guess that I start and stop Mozilla (or MozMail) 40-50 times
(once every ~15 minutes?) on a busy day.. and today has been a busy day.  I
wonder if this increases the chances of me closing Moz while some disk-based
mail operation is going on, and *that's* the actual source of the problem?

Wonder if mozilla going to the *birds will have any effect on my issues?  :)

    
    

    
  
Eric, that's certainly interesting.  However, we don't do much in the
background, and from your prefs.js, you're not even checking for/downloading new
mail in the background. One thing we are doing in the background is purging junk
mail on a 15 minute timer (in your case) and it's possible you could shut down
while that purging was going on. That shouldn't cause big problems, but it might
be a clue.

Re this comment:
"a newly arrived email overwrote an old email that was
in the Inbox, but had been moved away by a filter.  Since the new email was
shorter than the old one, reading the mailbox file makes it appear that the
ending of the old email is tacked onto the tail of the new message."

To me, this sounds like the mailbox truncation failed.  When we download pop3
mail and a message gets filtered, we move the message contents to the filter
destination folder, and we truncate the mailbox at the start of the filtered
message. Then, we write the next message at the end of the mailbox.  If the
truncation fails, all sorts of screwy things can happen. I don't know why the
truncation would fail, but I can put up an error message when that happens so
we'll know if that's what's happening to you.

    
    

    
  
Eric, do you have any message filters that move messages directly to the junk
folder? Can you send me your msgFilterRules.dat file?

    
    

    
  
Hi David,

With regards to the junk deletion, wouldn't a problem with timed deletes getting
cut off only affect the Junk box, since the mails are already long gone from the
Inbox by the time they get purged?

Your comments about truncation are interesting, but doesn't that imply that it's
only operating on messages at the tail of the inbox?  The message in question
was the very first one in the mailbox file.  So, my mailbox looked like this
(using my old standard 1k block paradigm) after the latest problem.  Ns are the
new message, and Os are the last blocks of an old message (that was filtered
days ago), and Xs are other messages in the inbox.  Fs are From: lines in the
mailbox file.

FNNNNNNOOOOFXXXXXXXXXXXFXXXXXXXXXFXXXXXXXFXXXX... out to 600kb

The O message was just long enough to have had its start at the head of the
file, before it was overwritten (10k).

I will attach the filters data file to this bug.  At one point I did indeed have
some filters that moved messages into Junk, but I think I deleted them.  IIRC
this was before Junk Tools could auto-move the messages for me (if there ever
was such a time? Can't remember.)  I know I got the first build I could get
after Bayesian filters went in.

    
    

    
  


  
There's a duplicate of the "Move to Folder" line on the second filter.	Might
that be (part of) the issue?

    
    

    
  
the duplicate "move to folder" line is almost certainly harmless although I
certainly wonder how it got there.

I don't really have a theory that would explain why messages towards the
beginning of your inbox would get corrupted. I have a guess, though - there's
some code that I'm not very happy about that allows you to mark messages read,
delete them, etc, in the inbox, while new messages are getting downloaded into
the inbox. I looked at it this morning, and I believe it could completely mess
up the file pos in the stream when downloading new messages. The other thing
about the truncate failing is that it implies someone else has the file open,
and that someone else could be wreaking havoc. I'll try to put in some
diagnostics to catch this.

    
    

    
  
Hi David,

I think you might be on the right track with the mailbox modification issue you
mentioned.  I know I used to have problems in the past if I read a message while
others were still downloading, in that my mailbox would explode in much the same
manner that it currently is.  However, ever since about moz 1.3 I've been very
careful to do nothing with the mail client while mail was downloading messages.

If you want, I can attach a copy of my Inbox file, which has just become messed
up again.  This time the .zip is only 100k.  

I've also noticed that the last six times I've had issues, there were no nulls
in the inbox.  Maybe something fixed in 1.5 corrected the nulls, but left the
truncation issue?

    
    

    
  
sure, why don't you e-mail me your INBOX (and the corresponding INBOX.msf file,
if you have it).

    
    

    
  
Hi David,

I sent you the file this morning, along with the .msf.  I realized after sending
that I didn't include the bug number in my message (doh).

In other news, I walked in this morning to 1800 emails (~1770 were spam) and had
no problems with the Inbox. Half hour later, I got 4 more messages (none spam
AFAIK) and had problems.  Lost two of the messages completely, and the other two
were fine.

    
    

    
  

      
      
      
      

        Attached patch
          
          
        proposed fix
        — 
          — Splinter Review
      

    


  
don't call filter plugins on open folders that are currently locked.

    
    

    
  
What Eric and I determined was that when one of his secondary e-mail accounts
finished downloading mail, we were running the junk filter on his primary
account, even if it was in the process of downloading mail. This could cause
various problems. The attached patch makes sure we seek to the end of the inbox
before writing messages, in case someone else has moved the file pointer, and
makes it so we don't run the junk filter on the folder open in the UI if that
folder is locked, as it would be if it was still getting new mail filtered into
it. There are still some potential problems, but I think this will make things a
lot better.

    
  

        Attachment #134335 -
        Flags: superreview+

    
    

    
  
fixed checked in, r/a=sspitzer over aim. Eric, you can try tomorrow's 1.6a build
and see if this problem is finally gone.
Status: ASSIGNED → RESOLVED
Closed: 21 years ago
Resolution: --- → FIXED

    
    

    
  
Bienvenu, you rock

    
    

    
  
I've observerd the same problems on Mox 1.4 and 1.5 Solaris 2.8 builds.

Erics and Davids observations seem to fit. I have message filters and the junk
filter to the same file. Sometimes messages where truncated (incomplete or empty
body) other times messages where merged, mostly into the header of another
message in the Inbox. Twice the inbox lost several mails at the end (even after
the 1.5 upgrade). These things where happening (approx 2 times a week) even when
I did not touch the MailNews window.

Can someone give a hint where/when the patch will be available? 
Is there 1.6a for Solaris (SUNs build - which does not require xprint) somewhere?


    
    

    
  
this fix went into 1.6a - I don't know about Solaris builds, but if there's a
1.6a Solaris build, it should have this fix.

    
    

    
  
Hi David,

Just checked the weekend's mail, and saw that the bayes filters on the Inbox
waited until after the primary account finished (therefore, weren't triggered by
secondary accounts finishing.)  No corruption encountered.

Don't know if I have the "authority" to mark this Verified (does the QA contact
have to do that?).. but it seems to me, based on the "Life Cycle of a Bug"
document, that this bug is ready for that stage.
Product: MailNews → Core
Product: Core → MailNews Core

    
  
See Also:  → 274330

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: