Open Bug 321371 Opened 19 years ago Updated 10 years ago

error message "error truncating the Inbox you may need to delete INBOX.msf" when messages filtered to folders

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

People

(Reporter: maxuuu, Unassigned)

References

(Blocks 1 open bug,
URL
)

Details

(Whiteboard: TURN OFF ANTIVIRUS SW and retest before reporting)

Attachments

(2 files)

dialog box showing error message
5.98 KB, image/png
Details
fix some of the bad symptoms of this problem
1.51 KB, patch
mscott
: superreview+
dveditz
: approval1.8.0.12+
Details | Diff | Splinter Review 
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12) Gecko/20050915
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12) Gecko/20050915

I get the error message, and then none of the mail matches the headings.  Some are blank, some from other senders.  The subjects are all wrong.  The error message comes up with every message received, and instead of sorting into my folders, some goes into the inbox with a different header, while the headers are in the correct folder with no messages attached.

Reproducible: Always

Steps to Reproduce:
1.Download mail
2.
3.

Actual Results:  
as above


downloaded email and sorted into defined folders
Do you use a virus checker for your incoming mails? Also did you try to do what it says, deleting the file Inbox.msf? Don't worry, your mails are in the file Inbox, Inbox.msf is just a generic summary file for the Inbox which will be regenerated (you might loose informations like Labels on your mails so, so just move Inbox.msf out of the way for now).
(In reply to comment #1)
> Do you use a virus checker for your incoming mails? Also did you try to do what
> it says, deleting the file Inbox.msf? Don't worry, your mails are in the file
> Inbox, Inbox.msf is just a generic summary file for the Inbox which will be
> regenerated (you might loose informations like Labels on your mails so, so just
> move Inbox.msf out of the way for now).
> 

(In reply to comment #1)
> Do you use a virus checker for your incoming mails? Also did you try to do what
> it says, deleting the file Inbox.msf? Don't worry, your mails are in the file
> Inbox, Inbox.msf is just a generic summary file for the Inbox which will be
> regenerated (you might loose informations like Labels on your mails so, so just
> move Inbox.msf out of the way for now).
> 

Yes, I use trend 2006.  I ran a complete scan, with no results.  I tried to find INBOX.msf but had no success.  I may have been looking in the wrong place.
See http://gemal.dk/mozilla/profile.html or http://www.mozilla.org/start/1.0/faq/profile.html#location. Also note the warning on the 2nd page that the folder is hidden, it might be that the Windows file search excludes files in hidden folders (i don't know for sure so).
Version: unspecified → 1.7 Branch
Please try to disable the Anti-Virus protection for the Mozilla profile, perhaps the AV program locks some of the mailbox files. See Bug 116443.
Severity: normal → major
I get this error occasionally too. I can always recover by deleting the MSF, but the error occasionally recurs. 

I will be filing a separate bug specifically for the way that this can result in a modal dialog popping up hundreds of times to finish checking your email. 

I have a hunch that Google Desktop Search is implicated -- perhaps it's touching Thunderbird's files at the same time as Thunderbird, causing the corruption. Don't yet have enough data to be sure. 

However, would strongly suggest Thunderbird should be able to recover from whatever transitory third-party visits (indexer/anti=virus/etc.) cause the corruption automatically and silently, with a retry or automatic MSF rebuild. 
this is a core bug
Assignee: general → bienvenu
Status: UNCONFIRMED → NEW
Component: General → MailNews: Backend
Ever confirmed: true
Product: Mozilla Application Suite → Core
Version: 1.7 Branch → Trunk
*** Bug 325491 has been marked as a duplicate of this bug. ***
Confirm same bug in OS/2 Seamonkey
I do not use any antivirus software
(In reply to comment #8)
> Confirm same bug in OS/2 Seamonkey
> I do not use any antivirus software
> 

This continues to plague me with:

Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:1.9a1) Gecko/20060510
MultiZilla/1.8.2.0f SeaMonkey/1.5a

It started a couple weeks ago while using the 20060210 build. I upgraded, thinking that this might help resolve the issue, but the problem has actually gotten worse. I am now forced to delete my Inbox.msf every few hours. I suspect it may be related to a corrupt msgFilterRules.dat file, and have rolled that back from a backup. If that proves to be the case, I will post both copies of the file here for consideration.

Lewis

PS - As this bug seems to be cross-platform, perhaps we (read: someone with authority - probably you, David) should change it from "Windows XP" to "All."
(In reply to comment #9)
> I suspect
> it may be related to a corrupt msgFilterRules.dat file, and have rolled that
> back from a backup. If that proves to be the case, I will post both copies of
> the file here for consideration.
> 
Scratch that. I had the same problem with a restored msgFilterRules.dat. I closed SM, removed both the old and the restored version of the file (thus disabling my filters), and even turned off Junk Mail controls. The problem persists.

I also tried removing my filter log file, which had grown in excess of 30MB. No difference there, either (this was before I restored the backup file).

I have created a new Inbox competely (closed SM, renamed Inbox to AInbox, deleted Inbox.msf, and let SM create a new Inbox file). Even a dozen messages were enough to cause it to mess up.

Extensions? I have tried this both with and without Mnenhy, which is the most "meddlesome" - for want of a better word - extension with regard to MailNews. No difference.

Anyone have any other thoughts?

Since I see this bug, too, (SM 1.0.1, Win2k) we can can exclude antivirus software, Google Desktop Search and hidden Windows folders, at least as sole reasons. Filters are part of the problem, because I'm, too, getting this problem when incoming messages are filtered into another folder. The only unusual thing (for me) I currently recognize is my large target folder (~260 MB) on my rather slow machine (AMD Athlon 500)...
When I look into the target mbox file (where in my case mails only go in and never move out), I do see stuff inside it where the new messages should be, but that content is not valid mail content, so just removing the msf simply can't help.
OS: Windows XP → All
Hardware: PC → All
Could this somehow be related to curroption in the XUL cache? Another glitch got me to thinking of that today.

In recent weeks, I've noticed considerable lag when switching between tabs, and high CPU utilization, too. I know we're talking about MailNews here, and I'm honestly not sure how much gets cached in XUL.mfl in the way of msgFilterRules.dat or anything else non-UI-specific (XUL cache is supposed to be *just* UI, isn't it?), however, I'm going to run for a few days without XUL cache enabled and see what I get.

Can anyone else subscribed to this bug tell me whether you have XUL cache enabled and the approximate size of XUL.mfl? Mine was ~6MB.

(BTW, even disabling XUL.mfl, I still see the file getting created - like in older versions. Is there a bug open for this?)

Lewis
Further comments (apologies for the bugspam; I'm just trying to document what I've been doing with this):

I moved all of my (hundreds?) mail folders (entire structure under Mail\) to a safe location. I then created top level directories (empty) for each of my POP3 and IMAP accounts.

So far, no problems whatsoever.

I migrated my existing (saved) Inbox over to the new structure by closing SM and copying Inbox as AInbox, and then restarting SM. I allowed SM to build a new .msf for AInbox, and then copied all the messages (7,081) to the new Inbox. Finally, I moved AInbox to Trash and emptied (I prefer to copy and then delete a large folder, just in case something goes wrong during a move operation - system hang, etc.)

Anyway, a couple restarts of SM, and still everything seems fine. I'll add some more folders back in later on, sans the .msf's. I'm starting to believe that the problem may be caused by non-Inbox indexes, which may not even be for filtered folders (as I've had the condition with and without filters).

More later.

Lewis

I've been adding folders back all day, on and off. No problems. I have not yet added filters back.

I had about a half dozen saved search (virtual) folders for current mailing list posts set up (as these show up in line after the system-created folders and before the other manually-created ones, so they're nice and visible). Upon migrating things back, I see these folders as empty. Perhaps something here was the cause of the trouble? Anyone else with saved search folders subscribed to this bug?

Lewis
I think I have it figured out, but I will need some confirmation from others seeing this behavior.

Subsequent to my comment #14, I did indeed experience the problem. I rolled bakc my testing once again, and started over. Among other things, I uninstalled Thunderbird New Mail Filters (which did indeed install - and work - in SeaMonkey Mail). Upon migrating all of my mail folders back, everything worked well until I copied msgFilterRules.dat back. Almost immediately, the symptom returned. I edited my filters and found several of them (perhaps corrupted by the extension, perhaps otherwise) filtering messages back to the mail account itself, insyead of to a legitimate folder. IOW, assuming two criteria were met, the action had the message getting filed in the account, but not in a folder. If one were to attempt to create such a rule, an error message would pop up, stating that the rule was not valid. However, as the rule was valid when it was created (i.e., it pointed to a real folder), the "sanity" check is not run against it subsequently.

I deleted outdated filters and either corrected or deleted the broken ones (about a half dozen in all). Since then, filtering has been working as expected.

The TB extension was Thunderbird NewFilters 0.1.1 by Ausdelicce (http://www.supportware.net/mozilla/#ext18). Was this the probelm? I have no idea. I'd like to hear from others on this bug, once they check their filter rules carefully.

If this was indeed the fault of a corrupt file, then this bug should be closed as INVALID.

Lewis
I do not have any extensions running on Seamonkey; I created 1 incoming mail filter (If <subject> then move to <folder>) via the dialog in-program. 

This bug occurs on a regular, but not consistant basis; some downloads, the delivery and sorting occur successfully; others, it's time for some delete-key fun inside Windows Explorer.

Added these comments to show that it may not be an extension issue.
I have just started getting this bug. This is first time this has happened. I am using SM1.0.2. I don't think I have any extensions. I used Moz 1.7.7 previously (still installed)with extension without a problem. I have 16 folders under the inbox and 28 filters. If any other information could prove useful let me know. Just before I posted this, I shut seamonkey down completely and restarted it went into mail and clicked the message button and emails came through OK. I am running popfile. I have been using it w/o problem for several years.   
I get this all the time in 1.5.0.6 on Windows XP. (I never get this in 1.5.0.5/Ubuntu or 1.0.xomething/FC4.)

I often have to quit, throw out my Inbox.msf, restart several times per day. It usually affects a small inbox which is checked by POP directly; it sometimes affects a large inbox which is checked by POP through POPFile (Spam filtering POP proxy).

Installed extensions are:
* Remove Duplicate Messages 0.1.02
* GMailUI 0.4
* Move Search Items 0.0.6
* Unselect Message 1.3
* No New Window on Double Click 0.2.3

The only other software I'm running that I would expect to touch Thunderbird's files is Google Desktop; however, the problem occurs even when Google Desktop is not running. 
 
I get a similar error message from Thunderbird version 1.5.0.7 (20060909) on WinXP, except that it includes extra information "after filtering a message to folder..." (see attached image)

I have been using Computer Associates EZ Antivirus with email filtering turned on. I have a high volume of spam that gets marked by spamassasin and moved to the "spam" folder by Thunderbird's message filters. I am also using Thunderbird's junk filter a lot now.

I usually delete inbox.msf as instructed, but the bug usually reoccurs a week or so later.
As an aside I had this happen again but as it has been some time since the previous problem I am not too worried, however what did happen after deleting the msf file it re downloaded all the messages that were still on the server. As I leave the messages on the server for about a week and I get about 70 msgs per day I ended up with about 500 duplicates downloaded
(In reply to comment #21)
> As an aside I had this happen again but as it has been some time since the
> previous problem I am not too worried, however what did happen after deleting
> the msf file it re downloaded all the messages that were still on the server.
> As I leave the messages on the server for about a week and I get about 70 msgs
> per day I ended up with about 500 duplicates downloaded
> 
Andy, this is most likely due to corruption of your popstate.dat file, and definitely not related to the deletion of your .msf. The popstate.dat file is responsible for tracking the most recently received message IDs. When this file is unreadable (or missing), SM rebuilds it, retrieving all messages left waiting on the server. So, if your setting is to leave messages on the server for 7 days, you will get the last 7 days' worth of messages again. Once this happens, of course, popstate.dat is once again in sync, so you should not receive them yet another time (unless something happens to popstate.dat again in the interim).

As a help, there is an extension available to delete duplicate messages. Simply right-click a message folder, and from the context menu, select the option to Delete duplicate messages. You may download (one of) the available extension(s) for this from http://split-s.blogspot.com/2005/02/duplicate-message-remover-01.html (this is the one I use).

Lewis
FWIW, bug 362612 has a fix for this message for OS/2 as of 2006-12-09, for 1.8.1 branch only, so it should get picked up in TB 2.0 - don't know about SM 1.1.

Max, what is your status?

Lewis, did you figure yours out yet?
Whiteboard: TURN OFF ANTIVIRUS SOFTWARE and retest before reporting
I regret that I am less computer literate than you all seem to be.

But in frustration I uninstalled Mozilla, and then reinstalled it.

I have had no problems since!
(In reply to comment #23)

> Lewis, did you figure yours out yet?
> 
I haven't seen this lately, Wayne. For some time, I was running an October, 2006 nightly (SM 1.5a), and currently, I'm running:

Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:1.9a2pre) Gecko/20061222 MultiZilla/1.8.3.0a SeaMonkey/1.5a

The only real issue I see with this build is that described in bug 357861. No msf corruption (yet), and filters seem to work as expected. That said, I've read the comments in bug 362612, and without looking at the code, I can't tell whether this may have any bearing on the 1.9 branch. I'll ask Peter to comment, however.

When experimenting with a POP3 account (that I don't normally use, it didn't occur with IMAP) I saw that the "Inbox" file in the profile was already opened by SeaMonkey before actually accessing the Inbox. So when the filtering process tried to open it exclusively for truncation, it failed. (The error message wrongly implies that something is wrong with the .msf file...)

That was with a 1.8 branch SeaMonkey on OS/2, so I don't know how that is related to other platforms. But one could easily test this with lsof on Linux or a similar tool on Windows.
For whatever it's worth, this issue occured spontaneously for me after upgrading from SeaMonkey release 1.0.6 -> 1.0.7.  No problems with this at all prior to that.

Sequence of events:
1) Install 1.0.7 over 1.0.6
2) On next start of mail, it spontaneously decided to re-build the .msf file for Inbox.
3) After the index rebuild, checking POP3 mail threw this error for each message that was supposed to be routed on receipt to a different mailbox, by filter rules.  The symptoms noted in the first post on this bug report were identical.

Deleting and re-building the .msf file did not solve the problem.  After much trial and error, I was eventually able to solve this by deleting the .msf, allowing it to re-build, and the doing a compact on Inbox _without_ fetching mail for the first time.  SeaMonkey must then be re-started, or the problem seems to recur.  Rebuilding the index, compacting, restarting and THEN checking mail seems to be the workaround here.

I have no extensions or plug-ins running on this system.  One potentially contributing factor is the ~2GB file size of my Inbox.  Otherwise, I have no explanation for why it spontaneously started occurring, nor why the above workaround seems to have resolved it.

I also experimented this bug recurrently. I must say most of the users don't want to deal with this problem. As you said, it's solved by deleting inbox.msf file (Or whatever msf file that correspond to the corrupt folder, but this issue always panic unexperimented users. I'm trying to convince my clients that mozilla y way better than outlook, but when they experiment problems like this is difficult to convince them about that.

I must add that this issue is recurrent for big mail folders. Most users don't erase or move mails, so they still in inbox making it bigger and bigger; in this conditions, this issue becomes really anoying. Today a client called me really scared because he's gone to travel and mail took 15 minutes to open. Of course, after writing the right msf file, everything went right, but my client asked me if it was possible to move back to outlook express?.

As long as I have installed mozilla thunerbird for more than 100 users, I can say that this issue have happen at least once for each of them.

Please, help me to prove that mozilla is far superior to other mail clients solving this bug.
Flags: blocking1.8.1.3?
Flags: blocking1.8.0.11?
Flags: blocking1.8.0.10?
Flags: blocking1.8.0.10?
if we fail to truncate the inbox, we don't want to tell the parser about the new offset - that throws things off.

I'd like to fix the underlying problem, but I think this should make things better in the meantime.
Attachment #257033 - Flags: superreview?(mscott)
Attachment #257033 - Flags: superreview?(mscott) → superreview+
I've checked that patch into the trunk and branch - it doesn't prevent the problem from happening but it should make the symptoms less severe.
David: should the patch land on the 1.8.0 branch as well?
Comment on attachment 257033 [details] [diff] [review]
fix some of the bad symptoms of this problem

yes, I think this would be good for the security branch.
Attachment #257033 - Flags: approval1.8.0.12?
Flags: blocking1.8.1.4?
Flags: blocking1.8.1.4+
Flags: blocking1.8.0.12?
Flags: blocking1.8.0.12+
Comment on attachment 257033 [details] [diff] [review]
fix some of the bad symptoms of this problem

approved for 1.8.0 branch, a=dveditz for drivers
Attachment #257033 - Flags: approval1.8.0.12? → approval1.8.0.12+
First Bugzilla post for me, but since I'm heavily using TB daily for a long time, and since this bug has become one of my best friends, hopefully can add some more useful info. I see this error message every day, so shutting down TB, deleting inbox.msf and letting it rebuild has become second nature ;-)

My observations:
- the ONLY factor of it happening for me is a stopped/lost Pop3-connection. I am currently travelling for half a year and have a very shaky GPRS connection in India...so whenever TB stops downloading my mail, and there was a message that was to be filtered out of Inbox into some subfolder, I get the error. 
- Example: 1. I check my mail by pop3; 2. TB is downloading 31 of 70 emails; 3. connection is lost, the little activity indicator stops spinning - I think by then 30 of the messages are there and can be seen, but #31 is there only partially, and thus not filtered out of the Inbox. It cannot be seen.; 4. I check mail again to get the rest - (HOWEVER, someone who is more bugzilla-savvy than me, pls file a new bug if this is necessary: often it's NOT possible to check mail again in this state. Clicking Get mail or doing it from the menu won't do anything, only solution then is restarting TB.) - it starts downloading with that last message, and THEN throws the inbox.msf error at me for every single message that should be filtered to some other folder until all messages are downloaded.; 5. Two options: either hitting OK on the error everytime, or killing the TB process and deleting inbox.msf right away. Doesnt really matter in which way it's done - result after the rebuild is that the messages that should have been filtered are now in the Inbox.
- A very interesting observation: if I stop a pop3 download manually with the stop button, this whole problem can be avoided. But there's a drawback: the last, partially downloaded message now lives in the inbox - even if to be filtered out - so eventually I have to rebuild inbox.msf anyway to see and delete those partial ones.

I hope this was helpful. I've long used TB on dialup in Germany and do some pretty heavy lifting with it email wise. And now in India the problem has multiplied because of the bad connection. I am very sure that that's the main factor, since I've seen it with all versions, even upgrading to 2 Beta2 didnt fix it - as I had hoped :-( Nonetheless, awesome program, keep up the good work, and if this one is finally fixed, will save me a LOT of time (I actually even keep the mail folder open at all times in my Explorer window so I can quickly delete inbox.msf).


I've landed this patch for 1.5.0.12 - since the patch doesn't "fix" the bug, I haven't added the 1.5.0.12 keyword - not sure how to get it out of the "needs landing" bug queries.
When I emptied my 10,000+ Junk folder, the bug apparently went away. (Before that, it happened almost daily.) Some speculations based on this fact:

As Veit Kuehne notes, the bug is triggered by some network connection timeout during the filtering of incoming mail. My guess is that the probability of this happening depends on the time needed for filtering, which in turn depends on the size of the folder we are filtering to. So to reproduce this bug, one should filter to a very large folder.
(In reply to comment #36)
> Some speculations based on this fact:
> 
> As Veit Kuehne notes, the bug is triggered by some network connection timeout
> during the filtering of incoming mail. My guess is that the probability of this
> happening depends on the time needed for filtering, which in turn depends on
> the size of the folder we are filtering to. So to reproduce this bug, one
> should filter to a very large folder.
> 
Or, as I have recently observed (yes, it's come back to haunt me), the symptom may appear when many filters exist for incoming mail; in essence, *anything* which may slow down processing of the folder thus holding the file open, per Peter's comment #26. I have in the past week or so added several more filters, and due to some other difficulties, thinned my Inbox down to 4,000 or so messages from 12,000 (note to self: *never* let those unfiled messages pile up like that again). However, while trimming the size of my Inbox file dramatically (naturally, I compacted after filing/deleting messages), the issue - which had not recurred for some time - reappeared, leaving me to believe that the size of the Inbox itself may have nothing whatsoever to do with the problem (and I am not negating anything you have said, Daniel, with regard to the size of the junk folder).

A few observations:

This isn't specifically related to anti-virus, as I see the problem on a box
that doesn't even have any anti-virus software installed.

Radically reducing the size of the Inbox doesn't seem to help.  I cut mine down from over 10,000 messages to about 2,000 and it didn't help with this problem.

I think there *may* be a correlation between the number of mail-filters and
the frequency of this bug.  I say this because I just subscribed to several new mailing lists and added filters for each of them (about 8-10 new filters, for a total of 60) and within a day or so of that is when I started having to delete my inbox.msf daily.  Before, I only encountered this problem once every few weeks or so.

The size of the Junk folder doesn't seem to be correlated either, from what I can tell.  I see this problem even if my Junk folder is empty or nearly empty.
I guess we're not going to get any additional fixes in  1.5.0.12 and there isn't going to be another 1.8-branch Thunderbird until rv:1.8.1.5 so removing blocking flags.
Flags: wanted1.8.1.x+
Flags: wanted1.8.0.x+
Flags: blocking1.8.1.5?
Flags: blocking1.8.1.4+
Flags: blocking1.8.0.12+
Flags: blocking1.8.0.13?
I've started experiencing this bug since moving to Tb 2.0.0.0. I can guarantee that it will happen for a particular daily message I receive -- report from ArcServe Backup software. I have a filter sending it to another folder. Everytime that message is filtered, up pops the message. The consequences I experience are:

- Download of mail stops so I have to "Get Mail" again.
- Sorting of junk mail fails meaning I have to mark the mail as junk.
- And obviously delete the msf file again.

The radical approach is delete the .msf file first thing in the morning, if I remember, compact the folder, perform get mail. Occasionally, I still see the dialogue box even after this procedure.

I currently have 30 filters in 3 mail profiles, by all accounts is not many. The Inbox is currently sitting at 40Mb for 1500 or so messages. From my experience it would appear the message / filing of messages is not only applicable to junk messages but overall response to mail filters. My assumption being junk filters are a special case of the normal mail filters. Correct me if I'm wrong.
Keith, how big is the daily message? Is your filter moving it to a unique folder or a folder that other filters on other accounts might be moving messages to? Are there any other messages that come in at the same time as this message? Or does this message come in when you start up the first time in the morning?

Do you have quarantining of incoming messages turned on? And do you have any virus checker installed that might be unhappy about this message?
(In reply to comment #42)
> Keith, how big is the daily message?
> 

44Kb according to the size column in Tb, or 3Kb if the backup failed. Today I made a conscious effort to make sure the Inbox was clean -- deleted the inbox.msf, opened mail, compacted the folder, closed Tb, deleted the inbox.msf, opened mail, get mail. Paranoia, eh?! Error dialogue appeared on this message, :-(.

> Is your filter moving it to a unique folder or a folder that other filters
> on other accounts might be moving messages to?
> 

There is one other filter that filters to the same folder. However, this filter occurs first, and is a boolean test of the body and fails in most cases. Should this first one be valid the message is tagged and moved, meaning the latter should fail because the message is no longer there. This first one only fires when I've forgotten to insert a backup tape, hence an error condition email message.

The latter fires almost daily and is simply a move of the message to the unique backup log folder.

The first filter is an 'IF from=backup server email address & body doesn't contain successful & to=my address, THEN tag important & move to backup folder'. The second filter is 'IF from=backup server email address THEN move to backup folder'.

> Are there any other messages that come in at the same time as this
> message? 
> 

No. There is only one message per day. Occasionally I've had firewall log messages bring up the error dialogue, however that's only when the backup one has already fired.

> Or does this message come in when you start up the first time in the
> morning?
> 

I use manual download of the mail, but this message fires the error dialogue whenever that message is in the queue for collection.

> Do you have quarantining of incoming messages turned on? And do you have any
> virus checker installed that might be unhappy about this message?
> 

No to both messages. Messages are scanned at the network boundary, and again on the internal server, so should be clean by the time we collect.

I'll disable the first of the filters and report back on the outcome, next Wednesday at the earliest.
I tend to see it more frequently when using an ISP other than my home.  Frequently happens with dial up connections.  Also happens with high speed connections in motels.
I disabled the second filter and not seen the error message since. Perhaps we need a "Stop processing rules" to avoid any additional processing if a rule has fired? This situation begs the question of whether the filter processing is asynchronous, or sequential. If sequential, then a stop processing filters would work. In my case I assumed the two filters never fired on the same message because they were mutually exclusive; that is, only one could be true, however, both pointed to the same folder. And it seems this is the underlying problem.

Is that the issue, that multiple filters cannot resolve to the same folder? I've not looked at the code so can't give an educated opinion, just a guess based on behaviour. Preventing multiple filters from resolving to the same folder would seem a little limiting. Anyone have clearer picture how the filters all work?
Keith, any time a filter moves a message, that stops the filter processing for that message. For that reason, sometimes people will set up an action that moves a message to the Inbox, i.e., the same folder, in order to stop filter processing for that message.

Filter execution is sequential, and in the case of pop3 messages, synchronous per msg - we evaluate the filter criteria for the first filter on a message, if it matches, we perform the action(s). If one of the actions is a move, we move on to the next message; otherwise, we move onto the next filter. All this happens before the junk filtering gets a chance to run.

These filters you're talking about are all on the same account, right? Can you e-mail me your filters, and perhaps the message that's causing the problem?
Flags: blocking1.8.1.5?
Flags: blocking1.8.0.13?
similar experiences
occurs while downloading with filtering - my inbox was over 2gb
emptied inbox item by item - still left massive 2gb garbage pile - problem still evident (have yet to try rebuild of .msf on the pile of garbage).
started fresh inbox - no further problems
Could prob be related to lack of garbage collection?
(In reply to comment #47)
> my inbox was over 2gb
> emptied inbox item by item - still left massive 2gb garbage pile - problem
> still evident (have yet to try rebuild of .msf on the pile of garbage).

It is *normal* for the message file to remain at the larger size until it is compacted. Deleting messages merely flags the records as deleted in the file and creates a copy in Trash. Compacting then reclaims the space used by those flagged messages.

This is surely a timing issue, and is exacerbated by multiple filters running against the inbox. As David points out in commment #46:

> Filter execution is sequential, and in the case of pop3 messages, synchronous
> per msg - we evaluate the filter criteria for the first filter on a message, > if it matches, we perform the action(s).

It would appear from the experiences of those reporting here that the corruption occurs more frequently when more filters are lined up in the queue and when more messages are either present in the Inbox or on the way down in a fetch. I have *never* seen this on IMAP folders, BTW; only POP3.

So, in short, the presence of a 2gb seemingly "empty" Inbox is not necessarily the presence of a garbage file, but rather a mail folder which has not yet been compacted. Compact the folder and then see how large it is (and if it remains large, that would probably be another bug).

Hello, yes I think this has something to do with multiple filters. I had over 14 filters running and would get this problem at least every other day (I delete the MSF file and it all works again). 

After reading this thread I deleted all my message filters and I have not had the problem for the last 3 days.

I am running version 2.0.0.4 (20070604)

I hope this helps.
This is a confirmed bug! However, there was no sign of that when I was working on TB 1.5. Now several day ago I upgraded to 2.0.0.6 and I keep getting this message since and on.

I don't have the database big enough but the message keeps on annoying me every time a new message is being filtered to anither folder. No matter if I do what it says - the next time it shows up again.
Keith, did David make progress with your filter file?

Ivan, what percentage of your messages get filtered to other folders?
Summary: error message "error truncating the Inbox you may need to delete INBOX.msf → error message "error truncating the Inbox you may need to delete INBOX.msf" when messages filtered to folders
Wayne, I would say near to 70% of all my messages are filtered to other folders. However it's not every filtered message that I get the bug box on. Sometimes it all goes smoothly - don't know why.

James in comment #49
> Hello, yes I think this has something to do with multiple filters. I had over
> 14 filters running and would get this problem at least every other day (I
> delete the MSF file and it all works again). 
> 
> After reading this thread I deleted all my message filters and I have not had
> the problem for the last 3 days.

James, what happens if you add filters back one by one?
PMFJI (and not to interrupt Wayne's query)...

I have 65 filters and see this on average once every couple of months. My brother has about three filters, and he probably sees it more often than I do (he also gets more spam than I do, so there's more Junk processing going on). We both are pulling mail from the same server, and often we are in the same office (though I have seen this even when I have been on the same LAN with the server).

Just wanted to throw that into the mix.

Presently, I'm running:

Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:1.8.1.5) Gecko/20070721 MultiZilla/1.8.3.3k eSeaMonkey/1.1.3

(The results are similar with this custom build as with standard builds, and have been consistent for some time.)

Many voters and cc here - can you devise a simplified testcase for your situation?       Example simplifications:
  * no junk filtering
  * no Google Desktop Search
  * stops by making a filter change, eg uncheck or change from body to subject


Bug 362612 has two interesting comments:
 1. "Windows seems...more lenient about opening files twice and things like that."
 2. "this is just a workaround"...and patch author seems to expect the problem to be resolved in this bug - do you feel 362612 is a workound

Lewis, did you encounter bug 362612?  Patched (as I read it) _false_ indication of truncation error for OS/2.
Whiteboard: TURN OFF ANTIVIRUS SOFTWARE and retest before reporting → TURN OFF ANTIVIRUS SW and retest before reporting
(In reply to comment #55)

<snip>

> Bug 362612 has two interesting comments:
>  1. "Windows seems...more lenient about opening files twice and things like
> that."
>  2. "this is just a workaround"...and patch author seems to expect the problem
> to be resolved in this bug - do you feel 362612 is a workound
> 
> Lewis, did you encounter bug 362612?  Patched (as I read it) _false_ indication
> of truncation error for OS/2.   
> 
It's hard to say, Wayne. I can tell you that I did see the *symptom* more frequently about a year or so ago, but I can't say definitively that the reason is tied into Peter's comments in bug 362612.

This could still be a filesystem-related issue, however, and the more filesystems we use, the greater the chance (obviously) for conflicting information. My installation, for example, is on a JFS volume. I don't know that I would see the same symptom on HPFS, NTFS, FAT32, Reiser, EXT3, etc., all OSes being equal.

I think your suggestion in comment #55, however, is a good one, i.e., creating a simple testcase with a standard set of variables (message filters on/off, junk filtering on/off, changing filters) and then trying to switch between filesystems.

I won't have a chance to set up anything like this for a couple weeks, due to the proximity in the calendar to Warpstock, but I will mark my list to get into it when I get back from Ontario.

Thanks for digging into this one. It's old, and a real nag. I'll do my best to help out as best I possibly can!

FYI, I still see this with 2.0.0.6 on NTFS. There is no anti-virus software enabled. I still get the error with Google Desktop disabled. 

I see the problem intermittently -- sometimes more than once per day, sometimes not for a week. It affects two different accounts which check mail via POP and have filters set up:

Account #1:
- checks POP via a local POPFile proxy
- "Enabled adaptive junk mail controls" is UNCHECKED
- "Trust junk mail headers set by [SpamAssassin]" is CHECKED
- has 15 filters, most of which move messages to other folders

Account #2:
- checks POP to a remote host directly
- "Enabled adaptive junk mail controls" is UNCHECKED
- "Trust junk mail headers set by [SpamAssassin]" is CHECKED
- has 6 filters, 4 of which move messages, 1 sets junk status, 1 deletes

There are also 2 accounts in the same client which have never seen the problem:

Account #3:
- checks POP to a remote host directly
- "Enabled adaptive junk mail controls" is CHECKED
- "Trust junk mail headers set by [SpamAssassin]" is UNCHECKED
- has 1 moving filter

Account #4:
- IMAP
- "Enabled adaptive junk mail controls" is CHECKED
- "Trust junk mail headers set by [SpamAssassin]" is UNCHECKED
- has no filters

As noted in related bug https://bugzilla.mozilla.org/show_bug.cgi?id=325491 -- I think that in addition to the core issue that soem file handling is astray there's the manner-of-error-presentation-issue: the excessive series of dialogs that barely allows even the manual 'delete-a-msf' recovery that is suggested. 

Followup to last comment: Yesterday I changed "Enable adaptive junk mail controls" to CHECKED on the two accounts that occasionally triggered the problem. I have seen the problem 3-4 times since then, far more often than usual. So while having this setting enabled is not necessary for intermittent occurrence, it subjectively seems to increase the frequency.  
re last several comments is there a "body" filter on every account that gets the error?
For my two problem accounts #1 (15 filters) and #2 (6 filters), none of the filters are 'body' filters.
(In reply to comment #59)
> re last several comments is there a "body" filter on every account that gets
> the error?
> 
Sorry for the delay, Wayne. Of my 65 filters, one - and only one - contains a "body" criterion (along with two others for that particular filter). However, I know for a fact that that filter was added long after I first experienced the problem.

On a related note, two days ago, I suffered a SeaMonkey crash while sending a message (it may have been downloading messages at the same time; I can't tell at this point (nothing reported in POPUPLOG.OS2 related to the crash, either).

Anyway, upon restarting SeaMonkey, I got our old friend, the dreaded "error truncating the inbox..." Wanting to get a snapshot of the Inbox itself for comparison purposes, I closed SM and copied it to a safe location. Then, in preparation for creating a new one, I deleted Inbox.msf. On a lark, I restarted SM, and allowed it to rebuild the index. Messages began to come down right away, so that time, it must have been the msf which indeed became damaged in the crash.

However, moments later, I got a hard TRAP (0003) in JFS, forcing a reboot. Which came first, the chicken or the egg? Did an impending JFS error cause the initial crash and subsequent msf corruption, or did the crash cause some trouble for JFS? Obviously, the timing of the two incidents was too close to be ignored (I rarely see JFS traps).

Gordon makes an excellent observation, as well: He sees the same issue with or without Junk filtering enabled, but more frequently with.

All anecdotal, to be sure, and I *will* follow up with more empirical data as soon as possible. I realize that it's simply not possible to debug an issue such as this without something more qualitative.

Today I was downloading email (using POP3) when my Norton AVS popped up an advisory message saying that a virus had been detected in an incoming message and then Thunderbird started to display the message "error truncating the Inbox you may need to delete INBOX.msf". Downloading halted about halfway through the messages. The symptoms thereafter were as others reported (empty messages when trying to display them etc.) So I closed down Thunderbird, deleted inbox.msf, restarted Thunderbird and to my horror found that my inbox was now empty and that hundreds of messages had disappeared (not just the new ones that had been downloaded, but also ones that had been there for months). This error is obviously very serious, but unfortunately not reproducible since I no longer have an inbox and I dont know what virus was detected by Norton AVS.
It sounds like Norton AVS deleted your Inbox file - is it possible that it put it into a quarantine directory?
Hi David

yes you are correct. On inspecting the Norton logs it tells me that it removed by inbox file. Nice!.  And it did not put it into quarantine. (BTW, the virus was Trojan.Dropper). My new Lenovo X.61s is only a few weeks old and the Norton Internet Security came pre-installed and pre-configured to take this action. So this should be a warning to anyone who thinks they do not need to check the default settings of pre-installed software.

Now its time to raise an issue with Norton.
My boss is having the same issue on TB 1.5.0.14 (I know, he needs to update, and we will shortly). I have not deleted the inbox.msf file yet, but from the above posts, that will only temporarily alleviate the problem, not solve the root issue. Has anyone found a true fix, or does the newest version of TB take care of it?

Lewis, did you ever get any verifiable, repeatable evidence?

Wayne, have you encountered any solutions elsewhere? 
Sorry for the late follow-up, Zach. I could have sworn that I posted this already, but don't see it...

I have not been able to reliably reproduce the problem, though upon re-reading the reports here (including my own), and referencing a recent recurrence of the problem, I'm of the belief that this - at least in some, of not most cases - is a symptom of an underlying filesystem issue or it follows on the heels of a bad shutdown of the app, leaving some open files (including Inbox.msf) in an unclosed state.

As a result, I think we need to look beyond some malfunctioning of the message retrieval process. What I am seeing is merely the true report that some damage has occurred, and SM's inability to properly index incoming messages. What we need to figure out is how to perhaps better safeguard the open files from corruption.

Just my latest observations, and apologies if they're not all that helpful.

I've been plagued by the "error truncating Inbox..." error.  I tried correcting filters, deleting .msf files, copying all messages to a subfolder and compacting folders.  Empty main Inbox file remained huge (2gig+) after compact.  Result....continued "Truncating" Errors.  I exited app, remaned inbox and inbox.msf to alternate name and copied in an empty inbox from another account.

Seems to work.

Mike
(In reply to comment #67)
> I've been plagued by the "error truncating Inbox..." error.  I tried correcting
> filters, deleting .msf files, copying all messages to a subfolder and
> compacting folders.  Empty main Inbox file remained huge (2gig+) after compact.

This seems significant to me, as this indicates that there is indeed something wrong with the Inbox file itself that it is not properly recognized. Some follow-up questions:

1. After deleting messages and compacting, were you able to actually view the contents of the Inbox?
2. Did you try simply deleting Inbox.msf, and letting the system rebuild the index?
3. If the answer to #2 is yes, were messages visible after the msf rebuilt?

>  Result....continued "Truncating" Errors.  I exited app, remaned inbox and
> inbox.msf to alternate name and copied in an empty inbox from another account.
> 
No need to copy Inbox from another account, as system folders (Inbox, Sent, Drafts, Unsent Messages, and Trash) will be recreated automatically upon startup if they are missing.

Thanks for the feedback on this, Mike.

Perhaps the best approach to resolving this is not to try to reack down the source of the Inbox corruption, but rather recognize the corruption and give the user the option to repair it (rename Inbox -> create new one -> move messages to new Inbox file -> delete damaged Inbox).

Anyone have any thoughts on my proposal?

>>1. After deleting messages and compacting, were you able to actually view the
contents of the Inbox?

  No... Folder appeared empty.  Messages copied fine to the new destination
  subfolder and were viewable from there.

2. Did you try simply deleting Inbox.msf, and letting the system rebuild the
index?

  Yes... Repeatedly.  I also deleted .msf of destination folders 
  defined in the filters.  After rebuild, messages were received 
  okay into inbox until processed by a filter.

3. If the answer to #2 is yes, were messages visible after the msf rebuilt?

   No.  Messages were not viewable in the folder from which they were
   deleted.  After rebuild of .msf and after compacting the original
   inbox remained at >2gig in size and messages were not viewable.

   Inbox was large...... and empty.

Mike

(In reply to comment #67)
> I've been plagued by the "error truncating Inbox..." error.  I tried correcting
> filters, deleting .msf files, copying all messages to a subfolder and
> compacting folders.  Empty main Inbox file remained huge (2gig+) after compact.

This seems significant to me, as this indicates that there is indeed something
wrong with the Inbox file itself that it is not properly recognized. Some
follow-up questions:

1. After deleting messages and compacting, were you able to actually view the
contents of the Inbox?
2. Did you try simply deleting Inbox.msf, and letting the system rebuild the
index?
3. If the answer to #2 is yes, were messages visible after the msf rebuilt?

>  Result....continued "Truncating" Errors.  I exited app, remaned inbox and
> inbox.msf to alternate name and copied in an empty inbox from another account.
> 
No need to copy Inbox from another account, as system folders (Inbox, Sent,
Drafts, Unsent Messages, and Trash) will be recreated automatically upon
startup if they are missing.

Thanks for the feedback on this, Mike.

Perhaps the best approach to resolving this is not to try to reack down the
source of the Inbox corruption, but rather recognize the corruption and give
the user the option to repair it (rename Inbox -> create new one -> move
messages to new Inbox file -> delete damaged Inbox).

Anyone have any thoughts on my proposal?

My thought on that proposal is that if it goes through, I will no longer regard Thunderbird or any other Mozilla product as safe or reliable, and will take every opportunity offered to declare it publicly. This is not only a serious nuisance, but a data-threatening bug that must be fixed.

If nothing else can be done, then, in the next general release, put in a call to Talkback that provides more information than the present stupid message that says little more than, "Something broke, but I'm not telling you what happened or where or how."
Mike: Thanks for the additional info.

John: I understand your thoughts, but if SeaMonkey or Thunderbird is not responsible for the file corruption (see my comment #66), we should at least have a way of dealing with the situation without having to instruct users to close the app, locate the profile, then the mail folders, rename Inbox, delete inbox.msf, start the app, move the messages back, and finally delete the renamed Inbox.

Your point about better error reporting is well taken, though Talkback doesn't run on all platforms (notably mine, OS/2), so we may still miss some valuable feedback on this.

Again, my reasoning is that we should have some internal mechanism for dealing with data file corruption (or someone might want to script an extension for it) to at least automate the recovery process.

Thanks for your thoughts.

I just had the joy of dismissing this dialog 71 times -- my usual workaround of drag the dialog so its 'close' is directly over the 'stop' toolbar button, and clicking twice in rapid succession, apparently doesn't work reliably while other slow POP fetches are also happening in parallel. 

And then, when only the dialog-popping mailbox was being checked, a few clicks in rapid succession atop the dialog-close/stop-toolbar crashed Thunderbird. That happens sometimes with this rapid-click workaround; I have to remember to only click twice. 

I keep a shortcut on my desktop to the mailboxes folder, so I can go through the "MSF" delete dance when this pops up, a couple times a week. 

Seriously, even if Thunderbird can't stop the 'error truncating' problem, and can't automate the perfunctory delete-and-rebuild of the the MSF file, can't the MODAL DIALOG be removed from a EVERY-NEW-MESSAGE loop that's IMPOSSIBLE-TO-CANCEL from the dialog? See also bug #325491.
confirmed on Thunderbird 2.0.0.14 (20080421)
David In reply to comment #29
> Created an attachment (id=257033) [details]
> fix some of the bad symptoms of this problem
> 
> if we fail to truncate the inbox, we don't want to tell the parser about the
> new offset - that throws things off.
> 
> I'd like to fix the underlying problem, but I think this should make things
> better in the meantime.

David, what's left?  Is there a workaround or way people who see this can reduce the likelihood they'll hit the problem?

A few other reports from the past year
http://forums.mozillazine.org/viewtopic.php?t=538847
http://forums.mozillazine.org/viewtopic.php?t=513245
http://forums.mozillazine.org/viewtopic.php?t=701765
Given that in this case (mine) "A little knowledge might be a dangerous thing"

The original bug might not be the same as the evolved discussion.
Several comment related to symptoms of this bug were noted/exacerbated after 2006-05-10

Bug 338549 and friends here bug 326273 and bug 381699 have shown that event that fired serially were no longer doing so.

Could it be that the underlying cause here is that events fired by filter hits have been affected by bug 326273 and stomping the database.



(In reply to comment #75)
> Given that in this case (mine) "A little knowledge might be a dangerous thing"
> 
> The original bug might not be the same as the evolved discussion.
> Several comment related to symptoms of this bug were noted/exacerbated after
> 2006-05-10
> 
> Bug 338549 and friends here bug 326273 and bug 381699 have shown that event
> that fired serially were no longer doing so.
> 
> Could it be that the underlying cause here is that events fired by filter hits
> have been affected by bug 326273 and stomping the database.

Is you conclusion partly based on this bug's setting version=trunk?  Bug 326273 and all it's fallout is *1.9* trunk.  But most or all of the people who have commented are 2.0 and 1.5 - what *was* trunk some years ago, but at gecko version 1.8 
Yep, I should have looked at the versions closer.

one of the things I don't like about setting version=trunk is you can't trust it after branches to mean, "affects only trunk".  

Perhaps version=trunk should disappear and be replaced by 
version=<somethingmorespecific>
(In reply to comment #78)
> one of the things I don't like about setting version=trunk is you can't trust
> it after branches to mean, "affects only trunk".  

Version=trunk means "affects trunk (possibly among others) at the moment I am writing". As long as it isn't FIXED, such a bug will affect all trunk versions, even if major releases happen in the meantime, unless a fix to some other bug happens to fix this one too as a side-effect.

When a bug affects both the trunk and a still actively supported branch, I recommend mentioning the fact on the Whiteboard, so that if and when the bug gets fixed, the assignee will (hopefully) consider whether it is worth porting to the Branch after the usual "cooking time".
QA Contact: general → backend
Product: Core → MailNews Core
Some comments refer to "larger than 2GB".
FYI.
Bug 449741 is for problem when all next conditions is met.
(1) Mail folder file size>2GB, (2) "quarantine option" is disabled, (3) Move of mail to other folder by message filter upon mail download(not manual "Run Filter").
No longer blocks: 449741
I have just encountered this bug on 2.0.0.19. I disabled my AVG incoming message scanner. That did not work. I tried deleting the .msf file. That did not work. Then I moved my inbox messages to a new folder and deleted the inbox folder. When I restarted Thunderbird a new Inbox folder was not created as some of the posts implied it would be. So I created a Inbox folder. The problem was still there.

Finally, I noticed that the folders that I was filtering messages to were very large even though they did not contain many messages (I use a search folder to delete anything over three months old). So I manually compacted the folders that were destination targets for message filters. The problem has now dissappeared for the time being.

This would appear to suggest that the variant of this problem that I was seeing is not related in any way to the inbox. But is solely a function of the uncompacted size of folders that are the destination target of filtering rules.
(In reply to comment #81)

> When I restarted Thunderbird a new Inbox folder was not created as some
> of the posts implied it would be.

Roger, I suggest that if you did not see a new Inbox folder being created that you perhaps renamed the wrong Inbox folder in the beginning (perhaps a Local folder one and not one specific to the account).

What I do for this is to actually rename my existing Inbox, start SeaMonkey (or Thunderbird), let it create a new Inbox (Inbox, Sent, Drafts, and Trash are all automatically-generated folders; if one does not exist, the code creates it upon startup), and then copy my messages to the new folder. Finally, I delete the old one.

I would also suggest a full chkdsk against the volume to rule out any underlying filesystem corruption.

Now, if the Inbox folder for that account *really* was not recreated, that's an entirely separate bug.

Cheers.
Quick follow-up:

Did you rename the folder from within Thunderbird? In that case, no; a new folder will not be created, as TB knows that Inbox is the new name. If you rename the folder from the filesystem, with TB closed, then it *should* create a new Inbox on next start.
(In reply to comment #82)
> (In reply to comment #81)
> > When I restarted Thunderbird a new Inbox folder was not created as some
> > of the posts implied it would be.
> Roger, I suggest that if you did not see a new Inbox folder being created that
> you perhaps renamed the wrong Inbox folder in the beginning (perhaps a Local
> folder one and not one specific to the account).
> What I do for this is to actually rename my existing Inbox, start SeaMonkey (or
> Thunderbird), let it create a new Inbox (Inbox, Sent, Drafts, and Trash are all
> automatically-generated folders; if one does not exist, the code creates it
> upon startup), and then copy my messages to the new folder. Finally, I delete
> the old one.
> I would also suggest a full chkdsk against the volume to rule out any
> underlying filesystem corruption.
> Now, if the Inbox folder for that account *really* was not recreated, that's an
> entirely separate bug.
> Cheers.

Lewis,

You are entirely correct. I do not have any account specific inboxes. Just a global one in local folders.

Roger
(In reply to comment #83)
> Quick follow-up:
> Did you rename the folder from within Thunderbird? In that case, no; a new
> folder will not be created, as TB knows that Inbox is the new name. If you
> rename the folder from the filesystem, with TB closed, then it *should* create
> a new Inbox on next start.

I think you got it right on your first comment. For information, what I did in Local folders was to create a subfolder called inboxbackup and move all the messages from the Inbox folder into that. I then exited Thunderbird and in the OS I deleted the inbox and inbox.msf files in the "Local Folders" directory in the profile. I then went back into Thunderbird and as the Local Folders tree did not display any Inbox folder. Also I could not receive mail, TB just did nothing, silently. Still within Thunderbird I created a Inbox folder within "Local Folders" I then could receive mail, but still was getting the "delete Inbox.msf" messages.

The real purpose of my post was to point out that in my case it looked like the error had nothing to do with the inbox at all. It disappeared as soon as I compacted the folders that were the destinations for automatically filtered messages. Some of these folders were physically very large, but none contained more than a hundred or so messages (due to my purging mechanism). I just thought that information  might be useful to you in tracking this problem down.

Cheers

Roger
Indeed, Roger, the more information we have, the better position in which we should find ourselves while tracking this thing.

I think that this bug will become a meta-bug for this symptom, which appears to have several different triggers, some of which are internal to the code, and some of which lie in the filesystem(s). Filtering is one of the usual suspects, as apparently when filtering a message and having difficulty getting the destination folder's msf updated (or getting the message into the folder in the first place), we throw the error quoted in this bug (even though it really may have nothing to do with Inbox.msf at all). We've also seen instances where Inbox (and perhaps Inbox.msf) are actually corrupted by failed filtering operations. We've also seen this issue when there have been no filters present at all, and perhaps the server was slow to respond(!)...

Apologies for the bug spam, folks, and have a good weekend.

Lewis
(In addition to comment #80)
> Some comments refer to "larger than 2GB".

Bug 450991(see also Bug 450359) may be one of causes of problem, if your have mail folder of 2GB<mail_folder_file_size<=4GB on MS Win. (both are "fixed1.8.1.18". i.e. fixed by Tb 2.0.0.18).
Check mail folder file size(Inbox, Junk, move/copy target folder of message filters) if problem occurs again.
Blocks: 325491
I am now having severe problems with this bug in Thunderbird 5.0. I get an error truncating. When I delete inbox.msf, and then restart, and compact, a duplicate of the message that was moved to the folder by a filter appears in my Inbox. If I run the filters on the inbox, I get two separate messages in the "move to" folder.

This is blocking an upgrade to Thunderbird 5, from 3.1.11. I cannot use the software if it's going to trash my inbox on a regular basis, and it does it with nearly every move because I filter almost every message to its own folder.

Any suggestions/workarounds?
Okay, I did some digging and have a workaround for this, but it's still not great.

I changed the scope of all the filters that move to folder to "Checking email (after classification)."

This seems to prevent any inbox corruption.

The side-effect is that now the new mail "star" only appears on the "Local Folders" heading, instead of appearing for each folder with a message moved to it. I prefer the other behavior, but not at a cost of a corrupt main inbox.

I should mention that all of the filters that cause problems are associated with the "Local Folders" account, rather than being associated with the individual POP3 accounts themselves. This in itself might have caused the problem.

Since the filters worked fine in 3.1.11, I would consider this a regression.
I would not be so quick to consider this a regression, as we still - even after all this time - are grappling to determine the underlying cause of the problem.

When I see this issue repeat itself in quick succession, I close the app (I use SeaMonkey, not TB), go to the file system, delete Inbox.msf, rename Inbox to AInbox, and restart the app, which automatically creates a new Inbox folder. I then move the messages from AInbox back to the new Inbox, and finally delete AInbox.

Thus, the corruption appears to be located in the Inbox itself, and not in the index file. That said, I have combed through a "damaged" one with a hex editor, looking for clues as to what might be amiss, and such things continue to elude my eye.

You might want to try creating a new Inbox and moving your messages, then changing your filters back. See if this clears the problem. I was actually heartened upon reading your Comment 88 that we might have a constantly-reproducible scenario, and could make some hay of it.

Cheers

Lewis
Just to add my report to this (not sure if it will add anything new, but you never know):

I upgraded to the PortableApps version of TB5 a few days ago. I use it on two machines - one running Windows XP and one running Windows 7 (64-bit). I collect mail from a number of accounts which all go to the global Inbox on Local Folders.

Generally speaking that Inbox folder remains empty because filters move incoming messages to other folders (which also exist within Local Folders).  Occasionally (because it doesn't match any of the filters) a message remains in the Inbox after mail collection but that's the exception and doesn't seem in any way related to the occurrence (or not) of the error.

I have AV scanning (F-PROT) in operation on both machines, but TB's /Profile folder (and sub-folders) is excluded from the scan.

This arrangement worked without issue on the previous version of TB.

Interestingly this error seems only to occur the first time I collect mail after starting TB (even if I ignore the advice to delete Inbox.msf).  Also (so far) if I wait for a few minutes after starting TB before collecting mail, I don't get the error (I have TB set up to collect mail every 10 minutes from each of my POP3 mailboxes but NOT to collect at startup).

As others have reported, when the problem occurs I get an error saying that there was an error truncating the Inbox file after moving a message and that I should shutdown TB, delete Inbox.msf and restart it.  I have done that a few times and (because the Inbox is generally empty anyway) have sometimes deleted the Inbox. (no extension) file, too.  Both are successfully recreated when I restart TB.
Since I updated to Thunderbird 5, I have the same problem.
The workaround that was best for me was to uncheck "Allow antivirus to quarantine individual incoming messages" in the Tools->Options dialogue, in the "Security" section's "Anti-Virus" tab. See bug #668592.

It's a bit of a UI mess, but it worked. Hope that helps.
(In reply to comment #93)
>See bug #668592.

Correction: bug #668952. Dyslexia strikes again.
I (for the first time) started getting this error the minute Thunderbird self upgraded... not happy.
I started using Thunderbird early in 2010 and never saw the problem until I upgraded from TB 3.11 to new TB 5.0 on my two machines using TB's pushbutton upgrade option.  On the laptop, the error happened immediately on the first getmail (but hadn't fetched email recently until that point).  I thought it was just a one-time format change due to the upgrade.  On the desktop (my main email), it was a few days later.  Then, of course, it happens regularly.

I have over a dozen POP3 accounts, with various filters on some (and a few filters on Local Folders.  The problem occurs regularly (but not consistently, not every time) on three specific filters on one account.  The filters are simple ones (if FROM="name", MOVE to "folder") and always worked fine previously.  (One filter actually scans for FROM="name" AND CONTENT="word" and sets TAG="tag" prior to the move.  If that fails, the second filter checks only FROM and moves.  Either filter could give the problem.)  Now, the filters usually work, sometimes for many emails on the same getmail (or for several successive getmails), or they may fail on a single email, all on a whim.

BTW, I thought column headers might be an issue if the INBOX ones are not the same as the destination box - but as I say, the filters work OK at one time and then fail the next.  (On INBOX, I set all from THREAD to DATE, plus SIZE and ACCOUNT.)
Further to my comment #96...
I'm using NORTON INTERNET SECURITY for email virus scanning on both machines.  Never had a problem with it as far as I can tell.
I've been getting this error too since upgrading to TB 5.0 a few weeks ago. Its very annoying to say the least.

I would agree that it seems to be related the message filtering. I use message filtering extensively to sort mail from 18 different POP3 accounts into various folders based on topic or action required.

One of my coworkers in my office, who also has several POP3 accounts that they poll, does not use message filtering and she does not get any such errors.

I am going to attempt R. Hanson's fix as outlined in Comment 89 above.

However, I have to agree that not having the folders flagged with new mail will be a serious pain in the butt to deal with, and an even more serious waste of time to find new mail.

This issue really needs to be fixed sooner rather than later!

Tom...
Same issue happening in TB6 (stable release) and TB9 (Shredder). Always occurs when some sort of message filtering is applied to the inbox. Deleting the filter rules "cures" the bug, but of course renders Thunderbird useless.

I will try Mr. Hansen's suggestion about disabling virus checking in TB (I'm using Avira Antivir 10 for info) and see what gives.
Yes, still buggy under 6.0.

I don't know if it can work for others than me, here is how I defeat this bug:
Walk through all your email accounts and disable the server setting "verify email at start" or something similar, it is just above the "verify email each X minutes", or something similar, I'm not English.

From what I've seen with my configuration:
- this error with truncating happens only at launch of Thunderbird, never in normal use (meaning 1 or 2mn after starting it...), 
- it happens after the message(s) have been moved, so nothing is lost.

My context is more than 10 emails accounts with POP, 3 with IMAP, RSS and News.
All pop accounts are retrieved in the global Local Folder, then filters are applied on it and parse emails between a bunch of subfolders under the global local folder. IMAP accounts are still retrieved at launch of Thunderbird without problem (no filter), as much as RSS (few filters) and News.

When testing I retrieve all accounts at once(POP, IMAP, News, RSS). It makes me the truncating error if I retrieve emails right after Thunderbird is launched, but not if I wait 1 or 2mn, which confirms my above statements without proving anything as I didn't test enough on different configs.


So, if a dev. read this, beginning with version 5.0, didn't you add a test or something else at the start of Thunderbird? On the Global Local Folder? On inbox? Something which lock the Inbox files?
Perhaps an anti-virus test if disabling it resolves the issue, but I won't test as my alternative solution works well.
I can pretty much confirm your observations Avva. I only get the error when I first start up Thunderbird and for me it is only with 1 specific account out of the 10 or so I have.
Sorry, I disagree.  I can run several GETMAILs (fetching over a dozen POP3 accounts) and have it work fine, including filtering on some.  Later, without stopping TB along the way, another GETMAIL (with a filter) will suddenly give the truncation error.  As Avva says, nothing is ever lost, but the error can happen at ANY time, not just right away.  I also agree that this started with TB 5.0 (immediately after upgrade from 3.1.11).
I disabled the quarantine option on the Anti-Virus tab, and the problem seems to be "fixed". My inbox index is not corrupted anymore after message filters have been applied.
Here's something out of the blue that might give a clue to the truncation problem...

I sometimes get a popup error stating that "The folder is being processed - please wait until processing is complete...".  I wonder if TB is perhaps tripping over itself somewhere (not "disabling interrupts" to cover critical update tasks that must start & complete without intrusion, such as reading/changing/updating pointers).  

I have several POP3 accounts.  I can force this error by running a second GETMAIL while GETMAIL is already running -- they collide a few times.  But the error happens spontaneously too, for no apparent reason.

Whatever allows a single series of sequential GETMAILs to trip over itself could also be putting something into the inbox at the same time it is trying to apply a filter and move it out again.  Hence a logic failure at best (two copies of the item, with pointers out of whack), or corruption at worst (garbled items the inbox or elsewhere, permanently mangled pointers, etc).
Personally I don't want to fix this issue by disabling a security feature, but it is interesting to know that it works for a lot of people.

Don't be sorry Dan Pernokis, it is this way that developers can find what's really wrong. Nevertheless we can't exclude that several facts can lead to this same error.
About the error you state "The folder is being processed - please wait until processing is complete...", it also can be forced by trying to compact the folder during email retrieval.
With that and the fact that disabling AV works for some, it also makes me think of a kind of (file) locking.

The errors in "Errors console" are as follow in an approximative english translation:
- "unattended end of file during the search for closing } of an invalid set of rules." (line 599)
- "Selector waited. Rules set ignored because of a bad selector." (line 599)
- "Analysis error of the value << filter >>. Declaration aborted. (line 526)
- "Analysis error of the value <<max-width >>. Declaration aborted. (line 333)

All theses errors refer to:
mailbox:///C|/users/you/AppData/Roaming/Thunderbird/profiles/blablayoutoo.default/Mail/Local%20Folders/Code?number=1603234

I know the "max-width" thing seems not related, but it appears at each iteration of a rules set that produce the truncating error, so it may be related or help to point out what make this happen.
I look at the filters and here is the shortest one that failed, I replaced characters a for an alpha, 0 for a number, for privacy:
name="Area"
enabled="yes"
type="17"
action="Move to folder"
actionValue="mailbox://nobody@Local%20Folders/NRL/Area"
condition="OR (from,contains,aaaaaa-aaaa.aaa) OR (from,is,aa.aaaaaaaa@aaaaaaaaaaaaaaaa.aa) OR (from,is,a000-aaa@aaaaaaa.aa) OR (from,is,aa.aaaaaaaaaa@aaaa.aa) OR (from,is,aa@0aa.aa)"

Nowhere in the msgFilterRules.dat there is { or }, it makes me think it is the thing that parse this file which don't like something inside this filter. But what... nothing seems special here.

In all case this join the fact it isn't -only- related to a start event.
Perhaps we can compare our error logs to see if the cause is the same?
Yep, still getting the same behavior in 6.0 here too. It does seem to be limited to the POP3 accounts that come into my Local Folders inbox that then go through a filter loop to move them to various sub folders based on either subject, the specific email account the message is addressed TO, or the email address that the message is FROM.

Tom...
I'm confused, today I don't have any pop box about a truncating error but my error console still gives me the errors I give in #105, so they probably aren't directly linked to this issue...
...And I DID have a truncating error for the usual filters, with only the same-old trivial warning messages.
I had never experienced this bug until I updated to Thunderbird 6 from the last stable version (I don't know what it was, but I always update Thunderbird when it asks).

I have been receiving this error ever since,  although I think it may be confined to one specific  folder I am filtering to (not sure unless I get email that triggers other filters)

Its really annoying.  Constantly present, I can close, delete the MSF, compact, the error still happens.

I don't know if its related either, or of this was something that happened earlier (I have no reason to believe it did) but since the updated I noticed some of my folders also got screwed up.   Some folders which I -know- had past messages sitting in them, are no completely blank,  and some folders were moved out of place to different locations in the Folder Tree.  I even had one folder  become a subfolder of the Trash folder.

It's really shaken my confidence in Thunderbird.  I'm not sure if I want to continue using it now..
Oddly for me the issue has completely gone away so far. I went into the accounts and unchecked in Thunderbird the checkbox for the program to check for new mail upon startup for each of my pop3 and imap accounts. That stopped the message to delete the inbox.msf file, but then I decided that I would prefer check at startup so I reversed this and tried the other option where I went into options and removed the checkmark to allow anti-virus clients to quarantine individual emails. After deciding that maybe I should have it checking for this, I decided to just live with the error message and have rechecked the box to allow anti-virus clients to quarantine emails. And even though it is back to the original settings before I messed with it, I haven't seen a error message pop up since. I did these changes over a couple day period and it has been about 2 days since I did the final switch back. I would normally see the message after starting up Thunderbird. 

If it doesn't stay this way I'll report back but so far so good. (and I've probably just jinxed myself)
Odd.

I don't check on startup, and I don't have Anti-Virus running,  but maybe I will give it a try, just ticking and unticking those settings.
I updated to TB 6.0.1 this morning.  The famous truncation error message still occurs.  No other settings have been changed.  (I have not applied the filter patch, and the problem only happens on filtered items.)
I had a new error message for a completely different folder than I have before today, so the unchecking and rechecking didn't seem to fix it. Previous to this I only had 1 folder that was giving this error. Today I got it with an entirely different folder that had never had it before. So ignore my possible fix, though it did do something to cause the error to move.

(In reply to Tim Norman from comment #110)
> Oddly for me the issue has completely gone away so far. I went into the
> accounts and unchecked in Thunderbird the checkbox for the program to check
> for new mail upon startup for each of my pop3 and imap accounts. That
> stopped the message to delete the inbox.msf file, but then I decided that I
> would prefer check at startup so I reversed this and tried the other option
> where I went into options and removed the checkmark to allow anti-virus
> clients to quarantine individual emails. After deciding that maybe I should
> have it checking for this, I decided to just live with the error message and
> have rechecked the box to allow anti-virus clients to quarantine emails. And
> even though it is back to the original settings before I messed with it, I
> haven't seen a error message pop up since. I did these changes over a couple
> day period and it has been about 2 days since I did the final switch back. I
> would normally see the message after starting up Thunderbird. 
> 
> If it doesn't stay this way I'll report back but so far so good. (and I've
> probably just jinxed myself)
I don't know..   I did the tick/untick thing and that seemed to fix it.  Or maybe it was something else I did?

It's been a while and I forgot to report back with what I did when the error resolved.
I have been looking for an answer on how to fix this since the upgrade to TB5.0.

I constantly get this error on startup.

However, I have noticed that I can normally download 1 message then I get this pop up error.  I clcik OK then this pop up idsappears then the rest can be downloaded.

I have tried various suggestions to fix this issue.
1. creating a new profile
2. creating a new inbox
3. compacting - after a massive clear out of old emails

I am now starting to think this issue will never be resolved and that I may have to look for a new email client (although I have loved TB since its creation & I do not want to change)

Please could someone get an answer to this
I tried something (I never knew was there) and I will report back in a few days.

I went to the inbox right clicked > properties > repair folder
This found messages which I read and deleted as they were old.

I then done this for every folder I have.

I am testing this as a solution for the next few days then I will get back to everyone.
This bug deals specifically with the error message regarding INBOX.msf. I seriously doubt that rebuilding all of your .msf's will make an impact. Also, as has been reported, recreating (rebuilding) the summary file *sometimes* works, but is not a cure-all for the problem.

In SeaMonkey, the option you have described is Properties | Rebuild Summary File, which is more descriptive of the actual function than TB's "Repair Folder" text, IMO, as there is really *nothing* performed on the mbox folder itself during this procedure, AFAIK (someone kindly correct me if I have misstated the operation).

Your Comment #115 also mentions creating an entirely new profile, which is also quite a bit of overkill. The next time you encounter the problem, I suggest following what I outlined in Comment #90, and see if that works for you.

As many people have reported, this issue may be related to the timing of message filters. If you have any filters, I would suggest turning them all off, and then turning them on again, perhaps one at a time or perhaps half of them at first, then a few more, etc. to see if that helps increase the time between happenings of this symptom. Please read through this bug report to get a better picture of what people have tried and reported.

Finally, it does absolutely no good, and only wastes electrons (and more importantly, time) to make statements such as your next to last paragraph in Comment #115. This is free software; if you use it, great. If you don't, that's up to you. None of us are getting paid to fix it (of whom I'm aware) nor to test it, nor to follow up on bug reports. Please be constructive in your comments, and you will help us *all* get a better handle on what is happening, why it seems to have gotten worse in recent builds (i.e., what is different *now* than six or nine months ago, both in terms of the app and in terms of your profile configuration). Many of us are frustrated by this bug, and many of us provide *paid* support for other people using this software, who in turn, complain to us about this issue. People with more understanding of the code than I (certainly) have been confounded by this thing for a long time, as it appears to be so random in its presentation. The more specific data we can gather concerning *when* it happens, the better the chance of finding the cause will be.

Cheers, all.

Lewis
The problem comes from a race condition between the filters and the quarantine option. The only 100% working fix so far is to disable the quarantine option in TB. Contrary to popular belief, disabling quarantine will NOT disable your antivirus nor its real-time scanner.
Lewis,

when I first looked into getting an answer to this problem, the most reported way of fixing it - even in mozilla's forums was to create a new profile.  Yes I agree it was overkill, but when it was the only sugestion I tried it.......

Replying that what I tried is overkill does not even help, but slightly infuriate me as I was trying solutions, not sitting on my **** waiting for it to stop!

I did read all of this thread but it was mainly going round in circles.

I have not taken off the TBs quarantine thing that was suggsted, but what I have done outlined above seems to have worked ..(for me anyway at the moment).

I really feel that your post regarding my post was rather elitest and I honestly don't think I would like to be part of this community if there are posts like that commming back
In addition, what I said about trying a new email client still stands.

your response to wehat I said is another that "only wastes electrons (and more importantly, time)"
In addition, what I said about trying a new email client still stands.

your response to what I said is another that "only wastes electrons (and more importantly, time)" is the exact same as you done
(In reply to ikthius from comment #115)
> I have been looking for an answer on how to fix this since the upgrade to
> TB5.0.
> 

There is a fix for similar bug (bug #668952) logged into Thunderbird 8. It may fix your problem. It fixed it for me.
Patrice, thanks for your concise presentation in Comment #118. Indeed, that's probably the best summary I've seen of this. That said, I think more things seem to trigger this (though the junk quarantine may be the major cause for the recent uptick in reports from users of recent TB releases). I believe (without re-reading all of the comments, here) that some have seen this crop up with and without junk handling turned on, and I know that one of my comments reported seeing this after a trap in JFS (thus, filesystem corruption). So, I would venture to say that the symptom/message presents itself after *some* latency in processing the inbox, either due to multiple filters and some contention with the junk filter or pretty much anything else which causes us to not get access when we think we should have it.

Is there a meta for all issues related to this "error truncating the inbox" thing?
I seriously doubt the conclusion of Lewis: the corrupted inbox.msf problem happens to me regularly since TB5 (and the checkbox "allow antivirus programs to quarantaine incoming messages" has been unchecked for a long time)

The problem with a corrupted inbox.msf still exists with TB8.
My config:
2 POP3 accounts (both using Global inbox in local folders), automatic mail and junk filters active, both POP3 accounts checked at startup, every once and a while the inbox.msf is corrupt and TB will hang on "building summary on inbox". Shutdown and startup doesn't help: TB seems to have no built-in recovery for corrupted .msf files. Only solution: delete inbox.msf.
Periodic deleting all .msf files and manual compacting all folders does not prevent this from happening. Turning off autocompacting doesn't help either.
Ruud: Have you run full diagnostics against your hardware and/or copied your profile to a different system to duplicate the behavior? What platform are you running? Is *anything* else running on the system which could be utilizing CPU or disk I/O, causing some latency to be introduced to the system?

You may seriously doubt my conclusion, but if you read over the comments in this bug, you should find that one thing which is fairly consistent is an interruption in processing between the mbox and the msf. Whether that is a hardware issue, a logical disk problem, antivirus, excessive filters firing, or some unrelated background process, we can't seem to tell for certain.

Before "seriously doubting" my conclusion (which I have not reached, BTW), you really need to run down the list of possible things which could be causing you to experience what we have noted in this thread.

Cheers
Hansen said in Comment 122 that a fix was coming in TB Version 8.  I installed 8.0 as soon as it was released and, sure enough, I have not seen the problem since.  I have made no changes to any settings -- in fact I've added MORE filters over time -- but everything seems to work fine now.  Thanks to all.

This is not to say the problem has been fully solved if it still occurs somewhere.  It may well be a latency problem (or even a file-locking issue) that has been tamed but not slayed by the latest fix.  I too had it suddenly appear in TB 5 -- why then?  Maybe it was always present but there were not enough trigger issues to make it show.  In my case, I have over a dozen POP3's being polled and I run several simple filters.  No problems now, but I used to get the error more than daily sometimes.

And just to expand on Lewis Comment 126 to Ruud:  As many will know, even a few simple processes can have a profound effect on latency in a lower-performance environment (low mHz, small RAM), mainly due to "overhead", whereas a slew of complex processes won't even make a fast system blink.  Things always run well until a critical threshold is crossed (such as one too many) -- and then all heck breaks loose.  On a slower machine, or on a faster machine with too much happening, the threshold comes sooner rather than later.
I'm running Earlybird 10.0 for OS/2 on eComStation 2.1 and I still have this problem (have had it for several previous releases).  The thing is despite the error message everything is filtered to the correct folder, so the message is useless.  Deleting all the .msf files doesn't change a thing either.  But I have to step through every error message.  Is there anyway to just turn off the message.  It is making TB almost unusable at this point, especially if I have a couple hundred emails to download.
Hi, Mark...

Have a quick read of my comment 90. AFAIK, that is the only way (presently) to deal with the endless barrage of these (bogus) error messages. Chances are that if you rename your Inbox, let TB create a new one, and then copy your messages (back) to it, you may just go for some time before seeing this again.

GL
Hey Lewis.  I forgot about this.  I did it a while ago and the problem crawled back under the rock it came from, but eventually, like Sarah Palin, it came back out again to annoy the hell out of me. ;-)  

It is rough getting old and forgetful.  Thanks for the reminder, but I wish TB weren't so brittle.  I try to never have more then 300 or 400 messages in my InBox, but I guess I will have to write this on my Thinkpad screen with a sharpie so I don't forget again.
Since I delayed the retrieval of half of my accounts (those which receive most emails), I don't have this problem anymore, for months now. 
But I'm a bit under the 100 emails at launch, if excluding RSS feeds where I also apply filters on, and which are retrieved at launch.

When the 1st email retrieval (at launch) is delayed to 10 minutes it is enough for TB to do its things and free the folders so that they can receive emails without prompt. Perhaps less is enough too, I didn't test as I don't need to retrieve more often my email accounts.
BTW, I now remember why I forgot about Lewis' suggestion.  It doesn't work here.  Still getting the same useless error message after renaming and recreating the InBox.
Assignee: dbienvenu → nobody
Wayne, was any of that content merged into https://getsatisfaction.com/mozilla_messaging/tags/error_truncating ? (Just noticed your last comment in the single bug with that tag, dated about a year ago.

This bug surely has become a classic, though I do see it less and less these days.

Mark, referring to your comment 132, perhaps the difference between us is that my workaround seems to do for SM, and you're on TB (I haven't tried on TB, as I don't use it enough to have bumped into this issue).
(In reply to Lewis Rosenthal from comment #134)
> Wayne, was any of that content merged into
> https://getsatisfaction.com/mozilla_messaging/tags/error_truncating ? (Just
> noticed your last comment in the single bug with that tag, dated about a
> year ago.

sorry, don't recall.  But bug 668952 was fixed for version 9
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: