931303 - filter fails (If "outdated msf condition" exists in "filter move target", & if "before Classification", error msg after fixing bug 782738, & skips "Move"). "The message could not be filtered to folder 'xxx' because writing to folder failed. Verify ..."

Status: NEW

(Thunderbird :: Filters, defect)

Description

[Craig Lassen]

Thunderbird has worked great for me until about two weeks ago. I am technically savvy and a very early adopter of Thunderbird, some fifteen years ago.

About two weeks ago I logged into Thunderbird, entered my master password, and received the following error message:
"The message could not be filtered to folder 'abc' because writing to folder failed. Verify that you have enough disk space, and that you have write privileges to the file system, then try again.

I checked disk space and privileges along with a half dozen other things before concluding that it may be something else. The interesting part is that once I have acknowledged all the message filter errors, the program will not error out again until I close the program and reopen it. In fact, as soon as I have acknowledged all the individual filter errors, the program will work fine filtering messages from that point forward. Additionally, in order to filter the messages that failed, the first thing I do is highlight the inbox and run "Tools->Run Filters On Folder". It always works and filters all the messages that failed upon opening.

Here is my configuration:
Application: Thunderbird 24.0.1
OS: Microsoft Windows 8.1 Pro Version 6.3.9600 Build 9600
System Type: x64-based PC

Comment 1

[Easy]

I confirm this bug exists in TB 24.1.0 and has started to occur a few weeks ago.
All behavior as described by Craig Lassen is duplicated here.
Running Windows 8.1 Preview, build 9431.

Comment 2

[dwgarrett]

This has been happening to me off and on for a couple of months. After update 24.1.1 it has become merciless. I have to kill TB because the message repeats too fast exit normally. It is not a write permissions problem.

Comment 3

[FordPrefect]

I too have this problem.  It started with upgrade to TB24.1.1. I am running on XP under Virtual Box.  My profile and folders are on a samba share from Fedora which is also running the VM.

In addition, the pop up window which would normally show the mail I have received, when I retrieve mail, is blank and cannot be closed.  I have to log off or reboot to clear it.

Comment 4

[The Beerslayer]

For whatever it's worth, I'm seeing this same problem, but only on one computer running Fedora 19 LXDE Spin (32-bit) with Thunderbird 24.1.0.  Another computer running Ubuntu 12.0.4 does not experience this issue.

Comment 5

[gero]

This report is a duplicate of #917769 -- can someone please change the status?

Comment 6

[gero]

(linkified: bug 917769)

Comment 7

[Hein]

I found this bug is very much alive I'm afraid.
It's been haunting me for months (at the very least) and I've tried every trick in an attempt to solve it, but all to no avail. Rebuilding the .msf files, rebuilding global sql, checking permission, checking disk space, creating new folders and adjusting the filter to that new folder, repairing the folders.

It only seems to occur when starting up Thunderbird and not after that initial check / sort. Firing the same fiter rule manually (after the start up) seems to work all the time with no issue whatsoever.
It seems to be the same set of folders that have the issue, but not every one of those folders, every time I start up Thunderbird, but I'm not 1005 sure of that, as I might actually not get mail to each of those folders /filters every time I start it up.

Using: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:24.0) Gecko/20100101 Thunderbird/24.2.0

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to Craig Lassen from comment #0)
> "The message could not be filtered to folder 'abc' because writing to folder
> failed. Verify that you have enough disk space, and that you have write
> privileges to the file system, then try again.

If this message was not shown in Tb release older than Tb 19, this is perhaps a result of "correct error check on MsgDBHdr write to MsgDB from Tb 19" in addition to existent "error check on mail data append to message store file".

> The message.
> http://mxr.mozilla.org/comm-central/source/mail/locales/en-US/chrome/messenger/messenger.properties#79
> This message is shown at.
> http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsParseMailbox.cpp#2509
> This line is in following code block.
> http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsParseMailbox.cpp#2483
> The message is shown in two cases.
> Error in Check (1) at 2491 rv = destMailDB->CopyHdrFromExistingHdr(nsMsgKey_None, mailHdr, true,
> Error in Check (2) at 2497 rv = AppendMsgFromStream(inputStream, newHdr, messageLength,
>       The message was for this Error in Check (2)
> Check (1) was added by Check (1).
> http://hg.mozilla.org/comm-central/annotate/f7e1fc4cf45b/mailnews/local/src/nsParseMailbox.cpp#l2491
> http://hg.mozilla.org/comm-central/diff/500529637889/mailnews/local/src/nsParseMailbox.cpp
> http://hg.mozilla.org/comm-central/log/f7e1fc4cf45b/mailnews/local/src/nsParseMailbox.cpp
> changeset id = 500529637889 is by Bug 782738
>   Bug 782738  Target Milestone: Thunderbird 19.0

Because error message was for Error in Check (2) for message store file write, it's confusing when Error occurs in Check (1) for MsgDBHdr write to MsgDB.

Before this Error Check (1) is introduced, following occurred.
 1. Write filter log of "moved to Target foldder by Rule-XXX",
    then executes actual "move mail to Target folder".
 2. "Write MsgDBHdr for moved mail to MsgDBHdr for Target folders" fails by some reason.
 3. Message filter appends mail to message store file for Target folder,
    even though MsgDBHdr is not correctly created in MsgDBHdr.
 4. Message filter removes the mail from Inbox.
 5. Because "MsgDBHdr for moved mail" is not written to MsgDB,
    mail moved by message filter is not shown at Target folder.
 6. Because "oudated msf condition" was generated in Target folder by above actons,
    Rebuild-Index(Repair Folder, Re-parse of message headr of all mail data) is invoked
    upon explicite folder open of Target folder.
 7. Because mail data itself is appeneded to message store file of Target folder,
    mail is normally shown the Rebuild-Index. 

If "Move target folder" is huge, user usually is aware of "Rebuild-Index upon folder click". But if not so large, user usually can't be aware of "invoking Rebuild-Index".

FYI.
When Error in Check (2) occurred during message filter move, "message filter move" silently failed with no error message. In this case, phenomenon is "filter move log(messae flter, Junk filter) was writen, but mail is still kept in Inbox, mail is not moved by filter". In this case, "manual filter run" usually works well always.
This is well known problem and pretty old problem.
"Show error message even when error in filter move" may be an improvement by Bug 782738.

Error in Check (1) can occur when;
(a) When Tb tries to open TargetFolder.msf file for filter move,
    someone interfered it, then Tb failed to open TargetFolder.msf file.
    Sojmeone : Other software. Other task in Tb.
(b) Other task locks MsgDB for TargetFlder.
    Then open MsgDB by message filter fails, then write MsgDBHdr to MsgDB fails.
    Then open MsgDB or write MsgDBHdr to MsgDB by message filter fails..
(c) Other task is using MsgDB for TargetFlder.
    Then owrite MsgDBHdr to MsgDB by message filter is interfered, and it fails.

If problem like (b), for example, conension with Compact Folder or Rebuild-Index, I think "other process is using..." like message was shown.
If problem frequently occurs "just after restart Tb", it's perhaps problem like (a), because MsgDB is not opened yet then .msf file is not opened yet.
If no other software, for example, virus scanner, auto-backup software, never accesses Tb's TargetFolder.msf file, this may be contension between "required many mail folder open after restart" and "TargetFolder.msf open for filter move after POP3 mail download which was invoked by new mail check upon startup".
If Global Inbox is used, it may be "contension of mail folder between/among multiple POP3 accounts".
If "message filter of multiple POP3 accounts" share same MoveTargetFolder, it may be "contension between/among multiple message filter moves".

Comment 9

[WADA:World Anti-bad-Duping Agency]

FYI.
Needless to say, as known by code, if "outdated msf condition" already exists in MoveTargetFolder before restart of Tb, phenomenon of "message filter move fails, and rebuild-index is invoked upon explicit folder open" consistently occurs.
Difference between "before Tb 19" and "Tb19 or newer" may be following, if "outdated msf condition" already exists in MoveTargetFolder before restart of Tb.
- before Tb 19 : 
  message filter move SILENTLY fails,
  and rebuild-index is automatically invoked upon explicit folder open.
- Tb 19 or newer : 
  message filter move fails, and error message is shown,
  and rebuild-index is automatically invoked upon explicit folder open.

I guess this is reason why many of problem reporters say "his problem was resolved by rebuild-index of mail folder(s)".
i.e. Somehow outdated msf condition already generated before restart of Tb 2x.y.z, and error message was shown by new Tb 2x.y.z when filter move failed, then outdated msf condition was cleared by Repair Folder(manually, or internally/automatically), then filter move worked again normally, thus error message was not shown.

To problem reporters who repeatedly/frequently see problem even after repeated Repair Folder of all relevant mail folders :

See bug 905576 for phenomenon of "repeatedly observed internal rebuild-index on local mail folder upon Search Folder open".
This is report on "local mail folder on SMB server", but I guess this is not "SMB file share" only phenomenon. I guess "SMB file share" merely eapands "timing hole".
In that bug's case, activity on mail folder looks "POP3 download + filter move" + "folder access by Virtual Folder after notification of new mail download or filter move of new mail" only, and "outdated msf condition" was not observed.
 outdated msf condition:
 Actual File size/Timestamp of file named FolderX upon folder open !=
 File size/Timestamp of file named FolderX which was saved in FolderX.msf upon folder close
It seems for me "Tb corrupts MsgDB data by himself".

Do you have special configuration or environment?
- Mail folder file on file server
- Pretty deep folder hierarchy
- Pretty many mail folders
- Pretty big mail folder(s)
- Mail folder sharing like Global Inbox
- Auto-Compact, with small threshold value, without enabling "confirmation dialog before
  start of auto-compact".- Daily use of Unified Folder View
- Periodical scan of Tb's file by virus-scanner
- Auto-backup of Tb's file
- Pretty many add-ons
- Special add-on, e.g. Lightning, Enigmail, ...
- Tb on USB memory(Portable Thunderbird"
- Power off of P without explicit termination of Thunderbird
- Power off of PC just after Tb window disappears
- and so on

Comment 10

[WADA:World Anti-bad-Duping Agency]

I looked into patch by Bug 782738. 
Difference between "before patch by Bug 782738(Tb 19)" and "after patch by Bug 782738(Tb 19 or newer)" was;
- before patch by Bug 782738(Tb 19) :
  Error check for (1) on "MsgDBHdr writing to MsgDB" was not executed.
  So, even when error occurred in "MsgDBHdr writing to MsgDB", error message was not shown.
  This is perhaps a reason of "message filter move silently fails".
- after patch by Bug 782738(Tb 19 or newer) :
  Error check for (1) on "MsgDBHdr writing to MsgDB" is done correctly.
  So, if error occurs when "MsgDBHdr writing to MsgDB", error message is correctly shown,
  although the error messsage is never appropriate for this error.
Error message is shown if Error in (2) "append mail data" occurs, regardless of "patch by Bug 782738 is applied or not".

Even if "outdated msf condition" already existed in MoveTargetFolder before restart of Tb 2x.y.z and the error message was shown after restart of Tb 2x.y.z, the "outdated msf condition" of the folder is cleared after Rebuild-Index by explicit folder open or manual Repair Folder. So, error is cleared.

Why Error in (1) or (2), perhaps in (1), so frequently/repeadtedly occurs on many folders in some users environment, even after repeated Repair Folder of many folders?
Almost all(NNN) folders have "outdated msf condition", and error occurs on several(M) folders per a restart of Tb, so "NNN/M restarts of Tb" is needed to invoke Repair Folder of all folders which already have outdated msf condition?

Comment 11

[seaofjoy]

I have started having the same problems with Thunderbird when I upgraded to Version 24.1.0.  I have upgraded to 24.2.0 and the problem continues.  I manage several email accounts via Thunderbird but the problem is only occurring on the first email account that is processed.  I really love the application and I will be willing to help troubleshoot the problem.  I am running Windows 7 Professional.

Jerry

Comment 12

[FordPrefect]

I no longer have the problem - I would say that it has gone away rather than being fixed.  I emptied the target folder(s) by moving all the mails in that folder to a new folder and compacted folders (don't know if I needed to compact but I did).

For information, size of folders went from 581359 bytes to 48869697 bytes.

Comment 13

[WADA:World Anti-bad-Duping Agency]

FYI.
I've opened bug 957495 for "misleading error message".

Comment 14

[Ray Lutz]

Because this issue is related to Bug 91776, I am cross-posting here, and describing my current work-around to these bugs.

I have had similar problems and have been pulling my hair out for a while. Again, this started recently, in the last update or two. I have been using Thunderbird for many years and am a sophisticated user.

There are a number of apparently interrelated problems.
1. all junk mail operations, message moving, etc. becomes nonfunctional after while. Restarting Thunderbird allows these functions to resume. This is apparently tied to getting messages, i.e. with automatic checking for messages every X minutes, then after that check, it locks up in this fashion. Also, after lock-up, the little icon in the tray of an envelope to indicate new messages will be present. Sometimes, multiple icons are shown, say five or six. Hovering over them and they will disappear.
2. When I turn off automatic message checking, then the lock up does not occur until I start checking messages. When I manually check messages, then I get the "Can't filter messages to mailbox XXX" notice. It only occurs during filtering when getting mail. Manual execution of filters does not do this.
3. Also, when the system "locks up" it is unable to save messages to the sent folder. I have turned off saving of sent messages and I just send myself a BCC. There is no way to save the message once it "locks up" as saving to drafts does not work either.

In terms of attempts to resolve this, I have done just about everything I know how to do, including deleting .msf files, deleting junk, moving messages to new folders, etc. etc. My latest test is turning off automatic checking for messages, and this delays the entry into the lockup mode. The only way out of the mode is to shut down Thunderbird and restart it.

I have a combination of POP accounts and IMAP accounts. The "Can't write to folder" happens when I check mail for a folder that has mail filters enabled during when mail is received. Thus, I can avoid the "Can't write to folder xxx" error if I turn on automatic mailbox checking but then it locks up after a while, and I have to restart Thunderbird. I'm not sure if it eventually locks up with manual mailbox checking.

------------

I've already been through repairing indexes, deleting all .msf files, etc. The reason I don't think it has anything to do with the folder indexes is because filtering works fine when I do it manually (i.e. run filters on folder). If I set all filters for a given inbox to not filter when messages are received, then after messages are received, I can manually "Run filters on folder" and it works fine. Thus it seems that the folder index cannot be the reason this fails.

I only suspect that the two issues are related because they both started at about the same time and seem to be contingent on the same set of circumstances. I acknowledge that bug 93103 is discussing the can't move messages issue. That is unresolved, but I found that by not automatically getting messages, I can avoid that failure at random times (i.e. when messages are retrieved) and instead the failure would occur when I manually invoke message checking. However, then I get this "Can't write to folder" error, which can be avoided by disabling automatic filters.

MY WORKAROUND:
Thus, my workaround for both problems is: (1) Disable automatic message checking and (2) disable automatic filtering when messages are retrieved. Instead, manually check messages on server (avoids folder write error) and manually invoke filters on folders (avoids "Can't write to folder" error). Also, (3) disable saving messages to SENT folder and simply request that BCC to sending address.

BTW, system runs on local disk to CPU (not network fileserver).
Message folders not particularly deep. About 16GB in Thunderbird folders.

Comment 15

[WADA:World Anti-bad-Duping Agency]

As I already wrote in comment #8 and comment #10, "where/why the error message is shown" is already clearly known. So, there no need to add comment of "Me too" or comment about "problem still occurs" any more.
Needed data for problem analysis by developer is;
a. What phenomenon internally happens when the error messsage is shown?
b. Why condition which produce the error message generated?
c. Even if error condition like "outdated msf" happened, internal rebuild-index is usually
   invoked upon "Folder Properties display" or "explicit folder open by folder click".
   Why "manual Repair Folder" is needed in many user's environment?
d. After explicit "manual Repair Folder" or internaly invoked "Rebuild Index",
   problem of this bug disappers in many user's evirnment.
   Why problem of this bug repeatedly occurs in some user's environment?

To all problem reprters in this bug :

When you see error message of this bug, do following, please.
(0) Call folder for which the error message was shown "FolderX".
    Check status bar message.
    Is message like "Rebuilding summary ..." shown at status bar?
    Check Error Console, Activity Manager.
    Is message for something wrong on FolderX seen? 
(1) Terminae Tb.
(2) Enable NSPR logging with MSDB:5, restart Tb.
    See bug 402793 comment #28 for NSPR logging.
    Win example : SET NSPR_LOG_MODULES=timestamp,MSGDB:5
(3) Writedown time, click FolderX at folder pane8explicit folder open)
    Is number of Unread message count updated at folder pane?
    Write down time, do "Repair Folder" of FolderX.
    How long does it take?
(4) Terminate Tb, keep backup of log file.
(5) Restart Tb without logging, and continue daily Tb use.
(6) Check log file contet by Text editor.
    Is log of "error opening db xxxxxxxx" seen? (xxxxxxxx is error code)
    Is log for "backup database" seen? (read bug 905576 for this log)
      log like "nsMsgDatabase::Open( ...\Temp\MozillaMailnews\<FolderName>.msf, ..."

To all problem reporters who experience this bug frequently/repeatedly even after manual "Repair Folder" of folders for which error message was shown :

Mechanism currently suspected;
() Problem like bug 905576 happens, and MsgDatabase(.msf) of "filter move target folder",
   call FolderX, is broken by Tb.
(ii) If FolderX is explicitely opened or FolderX is accessed by Virtual Folder,
   internal rebuild-index is invoked as seen in bug 905576, so FolderX is recovered.
(iii) Prior to Tb 24, "filter move" doesn't check "error return code of MsgDBHdr add".
   So, when MsgDatabase for FolderX is broken, Tb 17 appends mail data to file named
   FolderX without correct update of Msgdatabase. Then outdated msf condition is generated.
(iv) Tb has problem of bug 495760. So, the "outdated msf condition" is not automatically
   cleared by "filter move" of Tb 17.
   Because patch for bug 782738 is not landed on Tb 17, "error in MsDatabase" is not
   detected by "filter move" of Tb 17.
(v) Because patch for bug 782738 is landed on Tb 24, "error in MsDatabase" or "outdated
   msf condition" is properly detected by "filter move" of Tb 24, althouh error message is
   pretty misleaing(bug 957495).
If above asumption is not wrong, and if "Repair Folder" or "internal Rebuild-Index" is executed for FolderX, error on FolderX usually won't occur again until "MsgDatabase" is somehow broken again by Tb, or until "outdated msf" condition is generated again.
"Repeated filter move error on FolderX" means "MsgDatabase of FolderX is frequently damaged" or "actual Disk space/Disk Quota shortage repeatedly happens".
And, any problem reporter says that there is no "actual Disk space/Disk Quota shortage" in his environment.

Why can "broken MsgDatabse" or "outdated msf condition" be so frequently/repeatedly generated in your environment?

Do following, please.
(7) Enable NSPR logging(MSGDB:5), and use Tb as usual.
(8) Check log file size periodically.
    If log file size is very large, terminate Tb and keep backup of log file,
    restart Tb with NSPR logging enabled(MSGDB:5)
(9) If problem of this bug happened again, terminate Tb and keep backup of log file.
    Call folder for which the error message was show "FolderX".
(10) Restart Tb without logging, and continue daily Tb use.
(11) Check log file content of all backup of log by Text Editor
     Is "error opening db ..." seen for FolderX or other folder?
     Is log on "backup database" seen for FolderX or other folder?

Comment 17

[WADA:World Anti-bad-Duping Agency]

"Filter move failure" message of this bug may be shown when contention on "filter move target folder" occurs between "filter move" and "auto-compact".
If you enable auto-cmppact, surely enable "Show confirmation dialog before start of auto-compact", and reply "No" or "Cancel" to the dialog if needed.
  mail.prompt_purge_threshhold : true=auto-compact is Enabled, false=Disabled
  mail.purge_threshhold_mb     : Total size(in Mega bytes) of deleted mails in entire Tb.
  mail.purge.ask : false=dialog is not shown, auto-compact is executed without dialog.
                   true=dialog is shown before invoke auto-compact.
If mail.purge.ask is manually changed to false to true, "Restart of Tb" is mandatory to enable dialog again. If you check "do this automatically" or "don't ask again" at dialog, mail.purge.ask=false is set. Once false is set by checking it at the dialog, "manual change to true, restart Tb" is required to enable dialog again.
So, don't check "do this automatically" or "don't ask again" of the dialog, unless you intentionally accept "silent auto-compact invocation by Tb" even though you are frequently/repeatedly seeing error message of this bug in your environment.

"Rebuild-index"(=Repair Folder) can also produce contention on "filter move target folder" with "filter move".
"Internal rebuild-index" is automatically invoked when mail folder is explicitly opened(folder click at folder pane), if "outdated msf condition" exists in the "filter move target folder", or if you intentionally deleted .msf file(manual/intentional MsgDatabase corruption by you), or if .msf file is somehow deleted by Tb, or if "file size of .msf=0" is somehow produced by Tb.
If folder size is big, needless to say, "rebuild-index" takes long.
If you see "filter move failure" on a folder, call FolderX, measure "time needed to complete Repair Folder of FolderX". If it is long, reduce folder size(move mails to other folder), and execute Compact, and measure "Repair Folder time" again.

Comment 18

[Ray Lutz]

An update on my testing. I have reduced what I have disabled to just automatic filtering upon reception of mail. This has to be done on each filter. I have enabled saving to SENT mailbox and automatic mail reception, with no failure. I do not do auto-compacting.

As I am not familiar with the code, my hunch from these functional tests is that there is a separate asynchronous thread started when filtering is performed, and these threads are competing with resources, and I think they are getting into a resource deadlock.

At present, messages are being received but not automatically filtered. I have several identities and several phases of filters, and most are based on whether the address exist in address books. I now activate these manually, when there is nothing else going on. And, I activate them one at a time. I think that the current design of TB, each identity operate in parallel when downloading messages, and it also attempts to filter each message one at a time. It may be more efficient to process filtering in batch mode rather than a message-by-message mode, esp. if filtering is extensive and volume of mail large. But that is likely a redesign that is beyond any bug-fix discussion.

I suggest that if someone is familiar with the code, you look at the details regarding file locking and possible deadlocks. Some code is not releasing the locks I think.

Comment 19

[Hein]

My (new) findings on this issue.
I have about 20 mail accounts fetch mail (mostly pop) and about 300 filters among those accounts, just to let you know a little about the setup.

- I thought it might have something to do with certain boxes being imported from a windows install, so I recreated those boxes. doesn't make any difference
- When I start TB. it will, almost certainly, fail doing some of those filters on specific accounts / mailboxes, but not always on the same accounts/ boxes and not every time.
- When I do a manual filter per box, no issues.
- When I select all the accounts and run the filters, there are no issues either!
so it doesn't look as if it can't do all the filters at the same time, it just messes up when it's puling in new e-mail as well.

Just my 2 cents worth.

Hope this gets solved as I think I'm getting Carpal Tunnel Syndrome from all the mouse clicks I have to do to skip through those filter errors ;-)

Comment 20

[thebtm]

I have found a work around. located the msgFilterRules.dat file. (there may be many on your machine if you have lots of mailboxes) open in a text editor and delete the rules related to the folder have issues. then you should be able to open thunderbird

name="Rule Name"
enabled="yes"
type="17"
action="Move to folder"
actionValue="mailbox://Email address@server/Inbox/folder"
condition="AND (from,contains,data)"

Comment 21

[WADA:World Anti-bad-Duping Agency]

To all problem reporters:

Actual "filter move" operation is executed by Tb after Tb writes filter log of "a mail is (normally) moved by a filter rule to a move target folder".
What filter logs are writen by the download of new mail(s) for which "filter move failure" occurred?

Did "filter move to same move target folder for other account(s)" happen at same time when your "filter move failure" problem occurred on the account/folder/mail?

Comment 22

[Hein]

Ok, in a final final attempt I cleared my error logs and shut down and restarted tb as often as needed, until the dreaded erro came up. As the error log won't let you copy the texts themselves, here is a screenshot ...surprising errors if you ask me, but this seems to be why this 1 message that came in couldn't get sorted (yes, 1 message, so nothing to do with multiple message / sorts running passed each other or anything)

I hope this will clarify and get this sorted as I'm already looking at alternative for tb although I don't really want to.
http://www.heinkuenen.com/?attachment_id=121

Comment 23

[WADA:World Anti-bad-Duping Agency]

Log what I mentioned is "Filter log" which is written by message filter.
  MS Win example.
  C:\ ...\Thunderbird\Profiles\wkeci8t7.ZZZ\Mail\<ServerName>\filterlog.html
As timestamp is not written in filter log, following is needed to know mails downloaded at same time.
- Keep backup of filterlog.html of relevant accout.
- Keep backup of file named Inbox , "move target folder" on which filter mobe failure
  happened, "other move target folder" on which filter move was OK upon same mail download,
  and check timestamp of mails in "From - <timestamp data>" line(local time without
  Timezone string) by Text Editor.
  Timestamp in "From - ..." line is date/time when mail is downloaded from server by Tb.

Comment 24

[Hein]

Nah, sorry, I'm giving up.
Out with Tb today
Bye

Comment 25

[WADA:World Anti-bad-Duping Agency]

(In reply to Hein from comment #24)
> Bye

Here is bugzilla.mozilla.org to report bug of Tb to developers, and to provide required for problem analysis by developers in order to help deveoper's hard work for resolving Tb's actual bug.
Here is never place to simply/merely let developers to know "you also experienced problem of this bug" without providing required data for problem analysis by developers.
Hereis never support forum nor Help Center.

Thanks very much for your responce of "Bye" in this bug.

Comment 26

[Hein]

Dear Wada,

I know, but I've been trying to get this issue sorted (and yes, I Have tried many this as requested) for the last 6 months or so.
There is a moment that it's just not worth it anymore, trust me :-)

bye

Comment 27

[WADA:World Anti-bad-Duping Agency]

FYI.

"filter move failure" message of this bug is shown when error is returned at here.
> http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsParseMailbox.cpp#2491
> 2491     rv = destMailDB->CopyHdrFromExistingHdr(nsMsgKey_None, mailHdr, true,
> 2492                                             getter_AddRefs(newHdr));
Code of nsMsgDatabase::CopyHdrFromExistingHdr is;
> http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#3560
> 3560 NS_IMETHODIMP nsMsgDatabase::CopyHdrFromExistingHdr(nsMsgKey key, nsIMsgDBHdr *existingHdr, bool addHdrToDB, nsIMsgDBHdr **newHdr)
Cause is error while trying to create MsgDBHdr for "filter moved mail" in CopyHdrFromExistingHdr.

Please note following.
If error code returned at line 2491 is not checked, filter code moves mail data to move target folder without proper MsgDBHdr creation for the moved mail.
i.e. .msf data is corrupted by filter move.
     This means, if same condition existed in your environment,
     .msf file data was corrupted by Tb 17 without warning,
     and rebuild-index was silently executed upon next explicit folder open.
Notification of "Filter move failure" itself is not problem.
Data needed in this bug is never "Me too" comment.
Data needed in this bug is :
  When, with what condition, error occurs in CopyHdrFromExistingHdr.
  What kind of error occurred in CopyHdrFromExistingHdr when problem of this bug occurred.

Comment 28

[WADA:World Anti-bad-Duping Agency]

If "outdated msf" condition is somehow generated, or if "fileSize=ZERO(null xxx.msf file)" condition is somehow generated, "mail move by message filter" can silently produce known problem of bug 261419 and bug 495760.
Once problem of bug 261419/bug 495760 happens, "Repair Folder"(Rebuil-Index) is needed.
Setting depeency to these bugs for ease of problem analysis, tracking, and search.

Comment 29

[Wayne Mery (:wsmwk)]

wada, do you believe the increased rate of this problem is caused by bug 782738?

Comment 30

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #29)
> wada, do you believe the increased rate of this problem is caused by bug 782738?
I never believe so.

I believe;
- "Rate/possibility/probabilty/frequency/... of (a) 'something wrong' happens" is never
  altered by patch for bug 782738.
  "Patch for bug 782738" simply added "error return code check" step in "Tb's program code
  which is relivant to filter move".
- Until change by bug 782738(which introduced correct/valid/proper/right/good/clever/...
  return code check at a step in Tb's code),
  XXX.msf file had been/was silently corrupted/broken/damaged/... by Tb's "filter move".
  By change of bug 782738(which introduced correct/valid/proper/right/good/clever/...
  return code check at a step in Tb's code),
  "additional corruption of already damaged XXX.msf by Tb upon filter move" was
  successfully avoided.
- When the '(a) something wrong' happens, "Tb with patch of bug 782738" correctly/validly/
  properly/rightly/... detects such error and shows error messages,
  even though the error message is pretty misleading as I already reported to Bug 957495.
- "Actual culplit of this bug and some other bugs" is, I believe,
  "What produces the '(a) something wrong' when (b) 'unknown condition/problem' or 
  (c) 'other something wrong which is different from (a)' somehow happens.

I believe that "what we should do in this bug" is now finding "what is (a)", and finding "what is (b) or (c)", instead of merely adding "me too" comments nor merely adding complaints on "unexpected8and bad/wrong/misleading/confusing/...  message" which was newly shown by patch for bug 782738".

Comment 31

[Wayne Mery (:wsmwk)]

> instead of merely adding "me too" comments nor merely adding complaints on "unexpected8and bad/wrong/misleading/confusing/...  message" which was newly shown by patch for bug 782738".

I hope you will agree the users making these comments are trying to help.  I think most have been reasonable comments, especially given that the bug is still marked UNCONFIRMED.

How do you propose we determine "finding "what is (a)", and finding "what is (b) or (c)""??

And if this is highly reproducible (is it??) why would we not attempt to find the regression rhange where the rate increased?

Comment 32

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #31)
> How do you propose we determine "finding "what is (a)", and finding "what is
> (b) or (c)""??

For example, msgDatabase.summaryValid value check by user in daily use.

Following is already known.
- As already known by bug 261419 and bug 495760, Tb's message filter
  won't invoke Rebuild-Index when outdated msf condition exists.
  And, rebuild-index surely happens in some user's environment.
  Further, bug 917769 can occur if "message filter move" happens when outdated msf
  conditon exists.
- In above case, rebuild-index is automatically invoked upon explicit folder open.
  But rebuild-index will not be invoked in some situations, then manual "Repair Folder"
  is required to recover from error.
- This bug occurs at following step.
  1. msgDatabase open => 1-a: .msf open is invoked  1-b: cached msDatabase is used.
     As already well known by bug 261419 and bug 495760,
     "filter move" goes ahead even if "msgDatabase open2 fails.
  2. MsgDBHdr creation request
     => If error occurs at ths step:
        Tb 24 : error message is shown by bug 782738 and "filter move" fails == This bug
        Tb 17 : silently goes ahead even when error happes at this step
                => generate "bad msgDatabase" status
  3. Append mail data to msdStore file
     => If error occurs at this step:
        Both tb 17 and Tb 24: error message is shown and "filter move" fails.
Known Problem like bug 261419 and bug 495760 should be surely ruled out from this bug first.
"Problem after problem after ... after problem due to bug 261419 and bug 495760" should be ruled out first.
And, an easiest way to know existence of "outdated msf condition" is;
   access msgDatabase.summaryValid property via XPCOM from JavaScript code
   when msgdatabse is not cached yet
   => msgdatabase.summaryValid = false is returned if msf file size=0 is forced
   => exception occurs if outdated msf condition exists

See bug 905576 comment #191, please.

> And if this is highly reproducible (is it??)

"Repeatedly/frequently occurs in some user's environment only" is never equal to "Reproducible".
I believe that majority of users who experienced this bug is;
  During Tb 17 era, something bad was generated in msf file for N folders.
  During Tb 17 era, "message filter move" silently keeps outdated msf condition.
  After migration to Tb 24, this bug occurred on (N-m where 0<=m) folders.
  After Repair Folder of (N-m-q where 0<=q) folders, this bug doesn't occur again.

Comment 33

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #31)
> How do you propose we determine "finding "what is (a)", and finding "what is
> (b) or (c)""??

A possible scenario.
1. delete .msf file of many folders, and restart Tb.
2. because of "deleetd FolderX.msf", msgFolderX.msgDatabase.summaryValid=false is set.
3. after close of FolderX, "filter move" opens FolderX, and "outdated msf condition" is 
   raised for FolderX, and "filter move" silently apends mail data to msgStore of
   FolderX. So, "outdated maf condition" is kept by "filter move".
3. Upon FolderX is opened open by other than "filter move", for example, Virtual Folder,
   msgPurge, message copy/save, internal rebuild-index is invoked.
   If "filter move upon download" happens while rebuild-index is in rogress,
   following may occur;
      "filter move" ignores "msf open error" and goes next step of "msgDBDdr add".
      If Tb 17, Tb 17 silently ignores this error(error check is not executed by Tb 17),
      and mail data is appended to msgStore file => "outdate msf condition" is generated. 
      If Tb 24, error at this step is reported, and "filter move" fails.

Therefore, "manual .msf deletion" case should be completely ruled out from this bug.

Comment 34

[Wayne Mery (:wsmwk)]

(In reply to WADA from comment #32)
> "Repeatedly/frequently occurs in some user's environment only" is never
> equal to "Reproducible".
> I believe that majority of users who experienced this bug is;
>   During Tb 17 era, something bad was generated in msf file for N folders.
>   During Tb 17 era, "message filter move" silently keeps outdated msf
> condition.
>   After migration to Tb 24, this bug occurred on (N-m where 0<=m) folders.
>   After Repair Folder of (N-m-q where 0<=q) folders, this bug doesn't occur
> again.

"never" is surely too strong a word. 

My point was simple, a) if something changed to increase the frequency (do you deny something changed to increase the frequency?) and b) if it happens statistically frequent enough for a person (and we need only one person), then c) one CAN determine when with which daily build it started.

Comment 35

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #34)
> My point was simple, a) if something changed to increase the frequency (do
> you deny something changed to increase the frequency?) and b) if it happens
> statistically frequent enough for a person (and we need only one person),
> then c) one CAN determine when with which daily build it started.

If you think so, please proof your assumption by yourself, please.
"Who has to proof it" is apparantly you, instead of me.

Comment 36

[Craig Lassen]

Wada, et al.
You guys are probably outside my league when it comes to actually helping troubleshoot the code, but perhaps I have a few pieces of information that might help the situation.
1. I experience this problem every day like clockwork. If I understand what you and others are asking, I don't think the frequency of occurrence, or corruption to destination folders, is changing; i.e. the first derivative of the number of occurrence is almost zero. I believe the the number of emails that don't filter, is dependent solely on how many new email are delivered, and how many destination folders are corrupted.
2. The thousand foot view looks something like this:
New email arrives (pop or imap)
Is Email heading to a destination folder that will cause error (is it corrupt)?
    a.	No – Will filter normally. No errors.
    b.	Yes – Is new email heading to a destination folder that has already been tried during the current TB session?
        i.  No – TB will pop up error message and after acknowledgement will fail to filter.
        ii. Yes – No error message, email will not filter.
3. Although I don’t know why the destination folders are becoming corrupted (the real question), I know that sometimes they can be fixed and filtering can be restored by repairing the destination folder or deleting it and recreating it. If I have to delete and recreate the original folder, I can always move all the emails that was in the original “filter-error” folder into a new folder, and then delete the original and rename the new folder to the original.
4. I often get (maybe always get) the “Do you wish to compact all local and offline folders to save disk space?” the first time a new destination folder seems to corrupt and fail the filtering prcess. This seems in support of a possible combinatorial cause that you have mentioned before. I have stopped allowing compacting to occur at all for several weeks now and it has completely stops the problem of some of my very large folders being deleted (something that also started to occur about the same (maybe exactly the same) time the filtering issue started. While this is seemingly a different bug, it must be related somehow in my mind at least. Stopping compacting has also considerably slows the destruction of destination folders, although not eliminated it I don't believe.
5. This all started to happen for me around the beginning/middle of December. Not sure what update that corresponds to, but the time frame window of when it started is definitive.
6. I upgraded to Windows 8 in October, so I thought originally that was the cause. I am now unclear if that was a dominate factor or an adjunct and unrelated coincidence. Sounds like others on XP are having the same problem so perhaps not.
7. I often don't understand the nuances of the complicated short-hand diagnostic talk that occurs here on this site, but if someone can give me some basic instructions they want tested, I'd be happy to help. I'm a security engineer with extremely good logical diagnostic skills and I hate the idea of moving to a new email client. I have some C++ and basic knowledge so perhaps I could be useful to you all.
8. Thanks to all of you who are sinking your teeth into this puzzle. Your efforts are appreciated more than you know by those of us reliant on the app but without the means to affect change.
5.

Comment 37

[WADA:World Anti-bad-Duping Agency]

(In reply to Craig Lassen from comment #36)
>    i.  No – TB will pop up error message and after acknowledgement will fail to filter.

"Cause and result" is opposite, isn't it?
  Something bad happened at step 2
  => "filer move" failed to move mail to target folder => Tb 24 showed error message
  1. open msgDatabse
     Note: As already well known by other bugs,
           "filter move" goes ahead even if error occurs at this step.
  2. try to add MsgDBHdr for moved message to move target folder's msgDatabase.
     If Tb 24, return code is checked, and if error, stop filter move, show error message.
     If Tb 17, return code is not checked, so, even if error happened, Tb 17 goes ahead.
     i.e. if same situation as this bug happened, Tb 17 surely produced 
          "outdated msf" condition of local mail folder or broken .msf data.
  3. append mail data to msgStore file of move target folder
  4. remove mail from source folder

> Stopping compacting has also considerably slows the destruction of destination folders, (snip)

Some other peoples said so, and I already asked to check with "auto compact=disabled" several times if someone repeatedly experiences this bug.

It looks that you are fortunately a valuable people who can produce this bug repeatedly/frequently, even after "Repair Folder".
And, your report of "if without auto compact, frequency of this bug declines" is evidence of that Compact is surely relivant to this bug.

As I already wrote, if local mail folder, "outdated msf condition" will invoke "internal rebuild-index"(Repair Folder). Because rebuild-index is time-consuming job, it can interere "filter move upon mail download".
This is reason why I asked msgDatabase.summaryValid check.
See bug 905576 comment #190 and already refered bug 905576 comment #191, please.
Because "outdated msf condition" is perhaps irrelevant to IMAP folder, it may be useless for IMAP case. However, summaryValid=false may be relivant to IMAP case, and "outdated msf condition" may be applicable to IMAP Offline-Use=On folder. So, "automatic summaryValid check of IMAP folder" is also supported.

Comment 38

[WADA:World Anti-bad-Duping Agency]

FYI.
Problem when contention occurs between Compact and someone" itself is well known and old problem.
Following is example of (i) the someone=Rebuild-Index(Repair Folder) and (ii) the someone=Message filter move.
> bug 137210: Compact local folder doesn't compact if we are rebuilding msf file
> bug 139215: mail not filtered while compacting folders
See dependency tree for bug 498274 for other examples.

External phenomenon/symptom depends on (a) Tb's version, (b) mail folder type(IMAP or POP3, Offline-use=On/Off if IMAP, etc.), (c) who is the someone, (d) problem occurs at which step of the someone, and so on.
So, if "contention between Compact and filter move" case, there is no need to add comment of "frequency of this bug was reduced by disabling auto-compact" any more.
Required thing in this bug for Compact case is;
  To clearly state whether your problem(this bug) actually occurred in your environment
  due to "contention between filter move and auto-compact", or not.

Comment 39

[Craig Lassen]

Wada, thank you for the information you provided. I think it may have helped me to figured it out.

Whenever the filter fails to move a new email into a destination folder and produces the popup error message, all of the failed destination folders have several things in common. 
1.	They do not show the number of unread messages next to the folder name.
2.	They do not have the name of the folder "bolded" even if there are unread messages in that folder.
3.	If you click on any of the failed destination folders, they momentarily build a “Summary File” for the folder. This is different from “Compacting”. The more mail that is contained in that folder, the longer it will take to build the summary file. When it is finished, it returns the number of unread messages if there are any and bolds the file name.

Once I realized that all the failing destination folders needed to build a Summary File, I:
1. Expanded all of my mail folders
2. Started at the top of the folder list and used my “down arrow” key to land on every single folder with my cursor. You will see it say that it is Building this Summary File at the bottom left corner.
3. Wait for it to say "Done" in the bottom left hand corner before moving on to the next folder.
4. I unexpanded (re-collapsed) the folders list
5. I closed TB
6. When I reopened TB all worked normally and filtering forwarding worked.

Questions:
1.	Is there a way to rebuild all the summary files for all the folders at one time?
2.	If not, is there a way to expand and collapse all the folders easily?

Last Note: I may not have all this exactly perfectly figured out, but I know I am on the right track. Here are my main oppservartions.
1.	It seems that File Compacting does not guarantee rebuild the Summary File.
2.	It seems that the filtering routine does not properly run if the File Summary is out of date.
3.  The Filtering Routine is unable to proceeed unless the there is a current File Summary file.
4.  The Filtering Routine does not know to call the Build File Summary Rutine.

Hope that helps.
Craig

Comment 40

[WADA:World Anti-bad-Duping Agency]

(In reply to Craig Lassen from comment #39)
All phenomena you saw is symptom of known bug bug 261419 and bug 495760, except "error message of this bug".
(a) If "outdaed msf condition" happens on "filter move target folder" which is local mail folder, "filter move" appends mail data to the "filter move target folder" without updating .msf file correctly, and keeps status of "outdated msf condition".
(b) Known cases which generates "outdated msf condition".
(b-1) Interfere of Tb's xxx.msf file open by other software or Tb's other task.
  When "filter move" happens on "move target folder whose msf==xxx.msf", if folder(msgDatabase) xxx 
  is not opened yet, and if other software already opened xxx.msf file, Tb fails to open xxx.msf file.
  In this case, "filter move" continues filter move action, and append mail data to file named xxx,
  without updating xxx.msf correctly.
  => "outdated msf condition" is enerated by "filter move".
(b-2) xxx.msf deletion by Tb, which is usually Tb's bug.
(b-3) Manual/intentional xxx.msf deletion by user.

If "outdated msf condition" exists on folder xxx, and if folder xxx is not opened yet(xxx is closed. xxx.msf is not opened/read), Rebuil-Index is automatically invoked by following.
(i)   Explicit folder open of xxx by "Folder click" at folder pane
      Note: This is Tb's design/implementation inherited from Netscape/Mozilla Mail&News.
(ii)  Show "Folder Properties" of the xxx folder via folder context menu.
(iii) xxx folder open by Virtual Folder open who has xxx in his Search Target Folders(see bug 905576).
This mechanism itself works pretty well as designed/implemented, as seen in bug 905576.
(Note: (iii) is answer to your Question 1. "Is there a way to rebuild all the summary files for all the folders at one time?")

Problem in this bug are:
- Why "outdated msf condition" is re-generated so frequently/repeatedly in some user's environment
  even after repeated manual Repair Folder or automatic/internal Rebuld-Index?
- When(with what conditions), is error message of this bug shown by Tb?

"File hadle shortage" happens in your environment?

Comment 41

[WADA:World Anti-bad-Duping Agency]

(In reply to Craig Lassen from comment #39)
> 4.  The Filtering Routine does not know to call the Build File  Rutine.

This is cited issue by bug 261419(and bug 495760).
According to developer's comment in bug 261419, reason seems "it's neaerly impossible to invoke rebuild-index while doing download/filter move for newlr arrived POP3 mail".
(write mail data of a mail in file named Inbox, and move it to other file named xxx if filter move is requested)

Comment 42

[WADA:World Anti-bad-Duping Agency]

FYI.
As I wrote in bug 495760, "outdated msf condition" itself is normal in ordinal environment, although edge and rare case.
  1. MsgDBHdr for newly appened mail is generated, but it is done in memory.
  2. Append mail data to file named XXX. This is "physical file write".
  3. Power failure, System crash etc. => data in memory can't be physically written to XXX.msf file.
In such case, in order to re-create XXX.msf data(msgDatabase, MsgDB which holds meta data of each mail=MsgDBHdr), Tb invokes Rebuild-Index upon next explicit mail folder open.

Comment 43

[Craig Lassen]

In Reply To Comment 40,41,42

Thanks again for your help. I feel like I am always catching up with everyone else on this issue so I deeply appreciate your patience with me.

Here is what I know so far:
1) Something is deleting the .msf files. Not sure if it is TB or OS (Windows X).
2) Deletions occur regularly but no one as yet understands mechanism or is able to duplicate repeatability.
3) Deleted files are not moved to TB trash or OS trash. There seems to be no trace of them after deletion.
4) When deletions occurs, they seem to happen in clusters of files, possibly all of the files or folders within a specific sub-folder. Why specific subfolder is chosen also remains unclear.
5) Time frame between deletions is still not clear although, so far, it seems that deletions are not occurring during periods when TB is closed and unused.
6) I am working on a file monitoring program that will provide accurate logs with dates and time of deletions. I have initially limit this program to track selected subdirectories and selected .msf files.
7) In order to rebuild all of the .smf files for a given account I use the Edit->Find->Search Messages (Ctrl+Shift+F), Pick the Account you want to fix (name at top of list). Check box: "Search subfolders", Set: "Subject", "IS", and enter Random Sting "hd3^8h_djsh3" for search. Hit Search Icon. This method works well and is significantly easier than selecting individual folders manually. Thanks for the heads up.

Comment 44

[WADA:World Anti-bad-Duping Agency]

(In reply to Craig Lassen from comment #43)
> 7) In order to rebuild all of the .smf files for a given account I use the
> Edit->Find->Search Messages (Ctrl+Shift+F), (snip)

FYI.
"Virtual Folder" in following is "(saved) Search Folder". If Search Folder is pre-defined, you can do it by "Single click".
> (iii) xxx folder open by Virtual Folder open who has xxx in his Search Target Folders(see bug 905576).
> (Note: (iii) is answer to your Question 1. "Is there a way to rebuild all the summary files for all the folders at one time?")

Comment 45

[WADA:World Anti-bad-Duping Agency]

(In reply to Craig Lassen from comment #43)
> 6) I am working on a file monitoring program that will provide accurate logs with dates and time of deletions.
> I have initially limit this program to track selected subdirectories and selected .msf files.

External phenomenon of "Rebuild-Index repeatedly occurs on random folders or some specific folders" is pretty simlar to bug 905576, except (a) error message is shown in this bug but no error message is observed in bug 905576, (b) "deleted FolderName.msf" happens in your case bug but "deleted FolderName.msf" is not observed in bug 905576, and (c) local HDD file in your case but SMB1 shared file in bug 905576. Read bug 905576, please.
As I already wrote in comment #15, "NSPR logging with MSGB:5" may also help your problem anlysis, because event of "msgDatabase open/close" is logged by MSGDB:5.

Comment 46

[WADA:World Anti-bad-Duping Agency]

(In reply to Craig Lassen from comment #39)
> Whenever the filter fails to move a new email into a destination folder and produces the popup error message, 
> all of the failed destination folders have several things in common. 
> (snip)

Does it mean "When you saw Rebuild-Index of a folder named XXX, error message of this bug was always shown for the folder XXX before the Rebuild-Index occurred on the folder XXX"?

Comment 47

[WADA:World Anti-bad-Duping Agency]

I could pretty easily reproduce this bug...
This bug is merely a variant of phenomena of bug 261419(and bug 495760) which I repeatedly referred to.
This bug is rather an improvement of bug 261419(and bug 495760) upon pop3 mail download.
  When "filter move to folder XXX" occurs upon POP3 mail download,
  if "outdated msf condition" exists in folder named XXX,
    Tb 17 : "filter move" silently appends mail data to move target file named XXX,
            and removes mail from Inbox
            => Keep "outdated msf condition", and makes it worse.
    Tb 24 : "filter move" shows error message(although bad message and misleading),
            and don't move mail from Inbox to XXX.

[Steps to reproduce] == Same as bug 495760
1. While folder XXX is not opened(XXX.msf is not opened),
   append mail data at end of file named XXX.
   => force mismatch between 
            "actual fileSize of XXX/modified timestamp of XXX"
        and "last fileSize of XXX/modified timestamp of XXX" which is saved in XXX.msf
2. Check msgFolder.msgDatabase.summaryValid of XXX folder from JavaScript
   in order to confirm that "outdated msf condition" surely occurs.
   => Exception is raised with 0x80550005(== outdated msf).
   => My JavaScript code does do nothing(not invoke Rebuild-Index) when 0x80550005 is detected,
      "outdated msf condition" of XXX is kept.
   => "Folder Open"(MsgDB open) of XXX is not executed by msgDatabase.summaryValid access,
      because "outdated msf condition" exists.
3. Message filter : If Subject Begins With Test, Move to XXX
4. Put new mail, which has Subject starts with "Test", in POP3 server's Inbox.
   Because my ISP provides Web Access, IMAP access, POP3 access,
   "Copy a mail to Inbox via IMAP, then Get Msg via Tb's POP3 account" is easy. 
5. Get "New Msg"
   => Annoying but Error message of this bug is shown for each downloaded/filtered mail
   => Mail is not moved to folder XXX. Kept in Inbox of the POP3 account.

Because above case is problem caused by "outdated msf condition", and because Folder XXX is not opened by filter move(anyone can't open folder XXX due to "outdated msf condition"), Rebuild-Index is automatically invoked by "Explicit folder open(folder click at folder pane)", "Open folder XXX by Virtual Folder open which has XXX in Search Target Folder" etc.

If error message were "Outdated msf condition is detected. Do Repair Folder, please" like one, ...  

"Reuild-Index" is invoked automatically upon folder open after error message. And manual "Repair Folder" is also possible any time.
Why error message of this bug is so frequently/so repeatedly shown in some user's environment, even though user(or Tb) does do Rebuild-Index or Repair Folder after error, even though user never does manual delete of xxx.msf file?

Anyway, confirming.

Comment 48

[WADA:World Anti-bad-Duping Agency]

FYI.
If "filter move upon pop3 download" happens when "Deleted xxx.msf condition", phenomenon is different.
0. Number of mails in XXX = 10. All is Unread. Unread mail count at folder pane=10
1. When folder XXX is not opened, delete XXX.msf
2. Filter rule : If Subject begins with Test, move to XXX
3. Put 3 mail with subject=Test in POP3 Inbox
4. Get Msg
   => Because XXX.msf is deleted, XXX.msf is initialized by DBOpen requested by filter move.
      Just after DBOpen event is notified to MsgDB Event Listener,
      msgFolder.msgDataase.summaryValid=false is set.
   => "Filter move" ignores msgFolder.msgDataase.summaryValid==false after DBOpen.
      Because nohing is obtained from XXX.msf, this status=="no mails in folder XXX" for 
   ==> Because "what occurs" is "move 3 mails to XXX which has no mail in it" for filter move,
       3 new mails are moved to XXX folder normally,
       with neglecting existent 10 Unread mail data.
       => Unread count of XXX = 3 at folder pane
At this step, XXX folder's status == Normal, with 3 new mails, no deleted mails, for Tb.
So, XXX folder is normally accessed, except loss of existent 10 mails, and Rebuild-Index won't be invoked automatically.

5. Repair Folder => rebuild-index is execued,
   then 13 Unread mails are shown again, and expungedBytes etc. are also recovered.

This is already known phenomenon/issue in bug 495760.
This is main reason why I requested to surely rule out "manual .msf deletion" case from this bug. "Delete .msf file" is well known as "a way to force rebuild-index when broken .msf".
However, "manual deletion of .msf file by user" is never officially supported by Tb.
User should be responsible to force Tb to do rebuild-index, if user manually deleted .msf file.
User should be responsible on any problem before the "Repair Folder(rebuild-index) after manual delete .msf file" is actually executed.

Comment 49

[WADA:World Anti-bad-Duping Agency]

FYI.
Even after manual deletion of XXX.msf file, if someone, who keeps "msgFolder.msgDatabase.summaryValid==false status just after his DBopen", accessed the XXX folder first, msgFolder.msgDatabase.summaryValid==false is kept.
  My JavaScript code is an example.
    Delete XXX.msf => Check msgFolder.msgDatabase.summaryValid value
    => MsgDB is opened, with null XXX.msf content
    => Because MsgDatabase is initialized, msgFolder.msgDatabase.summaryValid==false is reurned
    My JavaScript code does do nothing when summaryValid==false is returned.
And ,if some other one, who can handle msgFolder.msgDatabase.summaryValid==false well, accessed folder next, Rebuild-Index will be invoked(for example, "Explicit Folder Open by folder click").
A prolem of "filter move" in this case is: 
  "Filter move" clears the msgFolder.msgDatabase.summaryValid==false status.

Comment 50

[WADA:World Anti-bad-Duping Agency]

(In reply to Craig Lassen from comment #43)
> Here is what I know so far:
> 1) Something is deleting the .msf files. Not sure if it is TB or OS (Windows X).
> 2) Deletions occur regularly but no one as yet understands mechanism or is
> able to duplicate repeatability.
> 3) Deleted files are not moved to TB trash or OS trash. There seems to be no
> trace of them after deletion.

If folder XXX is opened by someone in Tb after XXX.msf is deleted by someone else, msgFolder.msgDatabase.summaryValid==false is set by Tb's MsgDB open code.
This was confirmed by:
  Add MsDB Event Listener for folder XXX.
  In Event Listener, if DBOpen event, check msgFolder.msgDatabase.summaryValid value.
  Note:
  If msgFolder.msgDatabase.summaryValid is accessed by JavaScript code when "outdated msf condition"
  exists, "outdated msf condition" is notified to JavaScript code by "Exception with 0x80550005".
  And MsgDB is not opened(not cached) because "outdated msf condition" exists.

(A) "msgFolder.msgDatabase.summaryValid==false just after DBOpen event" was always observed when Repair Folder and Compact is executed. This is pretty normal, because Repair Folder/Compact re-initializes XXX.msf then creates correct XXX.msf content.
(B) "msgFolder.msgDatabase.summaryValid==false just after DBOpen event" was also observed by "Delete XXX.msf, then access msgFolder.msgDatabase.summaryValid by my JavaScript code". In this case, "Exception with 0x80550005" occurred. And the "Exception with 0x80550005" continued until Rebuild-Index is invoked by "Explicit Folder Open(folder click at folder pane)".
i.e. "outdated msf condition" is generated and kept as expected, if "folder with XXX.msf is deleted" is accessed by my JavaScript code first.
(C) "msgFolder.msgDatabase.summaryValid==false just after DBOpen event" was also observed by "Delete XXX.msf, then do Filter Move to folder XXX". However, after end of the Filter Move, msgFolder.msgDatabase.summaryValid==true was always returned to JavaScript code.
i.e. "Deleted msf condition" and "outdated msf condition" is cleared by Filter Move, if "Filter Move" accesses "folder with .msf is deleted" first.

Latest verson of my addon(enhanced for problem analysis of bug 905576 and this bug) supports this "msgFolder.msgDatabase.summaryValid==false check just after DBOpen event".
Craig Lassen, see following page, and check msgDatabase.summaryValid value, please.
http://www.h2.dion.ne.jp/~radon/mozilla/tb-addon/WinBackMyTrash-0-1-0.html
http://www.h2.dion.ne.jp/~radon/mozilla/tb-addon/WinBackMyTrash-0-1-0-Automatic-summaryValid-check.html

Comment 51

[WADA:World Anti-bad-Duping Agency]

(In reply to Craig Lassen from comment #0)
> Additionally, in order to filter the messages that failed,
> the first thing I do is highlight the inbox and run "Tools->Run Filters On Folder".
> It always works and filters all the messages that failed upon opening.

It was pretty normal phenomenon.
  When "outdated msf condition" still exists,
    If manual filter run(after the fact filter),
       Tb(Tb 24 too) silently ignores the "outdated msf condition",
       and appends mail data to move target folder, and closed folder.
  When "outdated msf condition" is already resolved by rebuild-index by explicit folder open,
    Any filter move by ant release of Tb work as expected.
Because "same procedure as manual filter move" is used if "Filter after Junk Classification", "Error message of this bug" is not shown when "Filter after Junk Classification". "outdated msf conditon" is silently kept by "filter move", if "outdated msf condition" exists.

"Error message of this bug" is shown only when "Filter before Junk Classification" and "filter move upon pop3 new mail download".
If "Filter before Junk Classification" is used, bug 762318(and bug 935934) occurs on both "mail which was moved by filter upon pop3 mail download" and "mail which was not moved due to this bug". Read bug 762318(and bug 935934), please.
If filter log is viewed, many many funny filter move log is seen. 
Even though many many "filter move log with no From:/Subject:Date:/Message-ID: data" surely exists in filter log, why no one refers to it in this bug?
No "problem reporter in this bug" checked filter log even though I asked to check filter log in this bug?
Problem of bug 762318(and bug 935934) won't occur in environment of "all problem reporters in this bug"?

Comment 52

[WADA:World Anti-bad-Duping Agency]

(In reply to Craig Lassen from comment #43 at 2014-03-01 12:36:33 PST)
> Here is what I know so far:
> 1) Something is deleting the .msf files. Not sure if it is TB or OS (Windows
> X).
> 2) Deletions occur regularly but no one as yet understands mechanism or is
> able to duplicate repeatability.
> 3) Deleted files are not moved to TB trash or OS trash. There seems to be no
> trace of them after deletion.
> 4) When deletions occurs, they seem to happen in clusters of files, possibly
> all of the files or folders within a specific sub-folder. Why specific
> subfolder is chosen also remains unclear.
> 5) Time frame between deletions is still not clear although, so far, it
> seems that deletions are not occurring during periods when TB is closed and
> unused.
> 6) I am working on a file monitoring program that will provide accurate logs
> with dates and time of deletions. I have initially limit this program to
> track selected subdirectories and selected .msf files.

Because you are a people who fortunately can see this bug's problem pretty frequently/repeatedly even after repeated manual Repair Folder and/or automatic Rebuild-Index by folder open, I believe that data of "Who(What process)/When deleted Tb's .msf file" is pretty easiliy obtained by simply getting "file monitoring data" by "file activity monitoring program" such as "Process Monitor for Win"(you are MS Win user).
Why no data is still provided by you, even after 3 weeks?

Comment 53

[Matthew Jurgens]

I have this problem as well. However this entry is not about "me too". For me this problem started happening after an upgrade to TB. I can not remember the exact version, 24 sounds about right. As far as I can work out someone changed something that started recognising when a problem occured with the filter moves and decided it was a good idea to pop a modal dialog box up about this condition. I think that this bug is the result of someone trying to apply a bandaid fix to another problem. I already had a problem which was always going to cause filter-driven folder moves. I didn't need a new dialog box telling me what I already knew about and making even more work for me. For me that other problem is being analysed in bug https://bugzilla.mozilla.org/show_bug.cgi?id=905576. For me if bug https://bugzilla.mozilla.org/show_bug.cgi?id=905576 was fixed I would probably never see this annoying dialog box.

The person that put this dialog box in or changed the trigger for it in TB 24 or thereabouts should simply remove it because it does not help anything but it clouds other issues.

I also suspect that multiple people contributing to this bug report are probably also victims of bug https://bugzilla.mozilla.org/show_bug.cgi?id=905576. 

Almost certainly affected by bug 905576 (mentioned that data and profiles are on a Samba share):
FordPrefect

Possibly affected by bug 905576 (some symptoms described):
The Beerslayer
Hein
Ray Lutz

If you are running data and profiles are on a Samba share you are at risk of experiencing bug 905576 and I believe effort should be focused on that bug which will lead to a root cause of both bug.

WADA does a great job trying to troubleshoot these bugs even coding up addons to provide more information but we really need someone who can look at and fix the the TB code. I think WADA has uncovered so much information that a developer should just start looking at the code now and fixing it. I'd be all in for test builds with a developer changing code to trying and lock down the root cause.

Comment 54

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #53)

Oh, you are seeing this bug too. (I've heard about itfrom you for the first time, even though I pointed this bug 12 times in bug 905576...)

As already wrote in this bug several times, "error message of this bug" is phenomenon from Tb 19 on which fix of bug 782738 was applied. See bug 957495 for problem of bad/wrong/misleading message.
Read comment #51 for phenomenon of this bug, please.
You look to have both "filter rule of Move To Folder with after Junk Classification" and "filter rule of Move To Folder with before Junk Classification".

As I already let you know by mail,
  "filter rule of Move To Folder with *after* Junk Classification" :
     when HdrAdded event is notified to MsgDB event listener, following is passed.
       messageOffset = Offset in move target folder,
       messageKey = messageOffset=Offset in move target folder,
  "filter rule of Move To Folder with *before* Junk Classification" :
     when HdrAdded event is notified to MsgDB event listener, following is passed.
       messageKey = Highest messageKey + 1
       messageOffset = Offset in Inbox
     This "messageOffset = Offset in Inbox" disappears after folder close at end of filter move.
  Difference comes from used mechanism.
    *after* Junk Classification : Same as after the fact filter == similar to manual mail move
       download all pop3 new mails to Inbox, apply Junk filter,
       then apply message filter to "all new mails which is not moved by Junk filter".
       This part is same as "Manual filter run".  
    *before* Junk Classification : Different from after the fact filter 
       (i) download a mail to Inbox, (ii) apply message filter to the mail.
       (iii) if move to folder is requested by filter, move to target folder,
             then remove mail data from Inbox(or temp file if quarantine option is On)
       (iv) repeat (1)/(ii)/(iii) for each not-downloaded-yet mail.

As I already wrote, if "outdated msf condition" already exists in filter move target folder, "error message of this bug" is shown and mail is not moved. This occurs only when *before* Junk Classification. This phenomenon is a variant of bug 261419(and bug 4957609) after "Filter before Junk Classification" was implemented and fix of bug 782738 was landed. Before fix of bug 782738, mail is silently moved without correct update of MsgDB(.msf). i.e. "corruption level of .msf" is raized by each filter move before fix of bug 782738.
And, bug 762318(and bug 935934) is also "Filter before Junk Classification" only problem.

If *after* Junk Classification, mail data is still silently moved to move target folder without correct MsgDB(.msf) update, and mail is removed from Inbox. This phenomenon itself is merely a dup of bug 261419(and bug 4957609).

Big difference between this bug and bug 905576.
 This bug: 
   As bug opener says, cause of pretty frequent/repeated error in bug opener's case is "someone
   deletes group of .msf files".
   "Filter before Junk Classification" only phenomenon.
   Major cause is;
     "filter move" silently ignores "qutdated error condition" upon folder open for "filter move".
     After fix of bug 782738, error code is checked upon MsgDBHdr generation for moved mail.
     Because folder open is not normally executed, something bad happens upon MsgDBHdr generation step.
     i.e. "Filter move" only problem.
     "Who/how/when generated cause of future 'rebuild-index'" is essentially irrelevant.
   Majority of problem reporters in this bug is perhaps;
     Outdated msf condition was generated on some folders during Tb 17 use.
     After upgrade to Tb 24, phenomenon of this bug happened on some folders.
     After manual Repair Folder or automatic rebuild-index by explcit folder open,
     problem of this bug disappeared. Even if happened again, it's pretty rare. 
 bug 905576:
   You quckly force rebuild-index by Virtual Folder open, even when "condition which produces
   rebuild-index" was somehow generated in some folders.
   Even though it, MSGDB:5 log of "error opening db 80550006" is repeatedly written for BackupDatabase.
   If problem only in "filter move", I believe "outdated .msf condition" should be generated
   by each "filter move upon pop3 download". However, it's not.
   So, never "filter move only problem".
   There is no problem in "Rebuild-index after something bad" itself.
   "Reuild-Index" itself is executed as designed/implemented, because "rebuld-index" is for
   "automatic .msf recreation mechanism in case of bad situation in .msf".
   Problem occurs in both "Filter before Junk Classification" case and
   "Filter after Junk Classification" case. It's merely that "phenomenon of this bug" is observed
   when 'condition what produces rebuild-index' is somehow generated.
   Problem is "When/Who/How generated 'condition what produces rebuild-index'".
   But nothing about "When/Who/How generated 'condition what produces rebuild-index'" is still known.
   We could merely recently see "msgDatabase.summaryValid=false" only one time, on only one opened
   folder, after "filter move" followed by "immediate close after filter move, then immediate open by
   someone(perhaps Gloda)".

   Which source have developers to check at this stage? All filter move relevant code?
   All folder open/close relevant code? All Gloda code? All Virtual Folder/Unified Folder related code?
   All "folder close upon shutdown" code? All ".msf file update" code?

   As known by analysis of data you provided, following is already known.
     By rebuild-index, "MsgDBHdr part in .msf file which was added by filter move" is not touched.
     Merely "Data writeen upon folder close" is appended to .msf file, and adding of a relatively
     big block(perhaps for Thread Pane Column selection) is seen in some cases.
     "Full read of .msf file data" is always executed by folder open.
     So, problem is only "excess full read of msgStore file" which you don't want to see.
   Why priority of such problem, which is problem since Tb 3, should be pretty high?

Comment 55

[WADA:World Anti-bad-Duping Agency]

FYI.
See attached data to bug 935934 comment #7 ( attachment 8395180 [details] ) for difference between "after Classification" and "before Classification" which can be visible by log data.

Comment 56

[WADA:World Anti-bad-Duping Agency]

About "Disablibg Auto-Compact" for troubleshootng of this bug.

As known by data attached to bug 935934 comment #7 ( attachment 8395180 [details] ), "Filter after Junk Classification" is same as "Download POP3 mail to Inbox + Manual run of filter on Inbox", from perspective of "Content in file named Inbox for POP3 account".
i.e. Mail data in Inbox is merely marked as Deleted when mail is moved by message filter to oter folder.
So, if "Filter after Junk Classification" is also used in addition to "Filter before Junk Classification", and if Compact is not executed, file size of file named Inbox will silently increase until file size exceeds file size limit(currently 4GB).
Please note that "Disablibg Auto-Compact" is for troubleshootng purpose. Do manual Compact periodically if you disable Auto-Compact for troubleshooting, please.

Comment 57

[Craig Lassen]

Wada, Et Al,

Thanks again for all your effort on this particular bug, I thought your most recent (and most in-depth) description/explanation of the bug was the best description by anyone to-date. 

I have not yet finished figuring out the high-level mechanism of when and how the msf files are getting corrupted and deleted. However, preliminarily, I ran across a couple of things that might be food for thought for those of you that know a lot more about the inner working of TB than I. Please let me know if these post are not helpful to you.

1. I was wrong about all the msf files being deleted. Only some are deleted. Here is some early data about the pattern so far:
     1.1.   None of the .msf files that were deleted were off the root account folder (i.e. Mail\AccountName\No_Deleted.msf )
     1.2.   Usually the deleted msf files were contained at least one level deep (i.e. Mail\AccountName\Folder_Any\Deleted.msf)

2. Most of the rest of the problematic msf files, actually exist, but they are zero length files (their content had been deleted). Here is the pattern so far:
     2.1.   Off Account Root, mixed - some msf files zeroed, some msf files okay (i.e. Mail\AccountName\Mixed.msf)
     2.2.   If the Subfolder.msf files in 2.1 above is zero length, then all of the msf files in that sub-directory will be zero length as well. (i.e. if Mail\AccountName\Folder1.msf is zero length than Mail\AccountName\Folder1\AllFiles.msf will be zero length as well).

3. In writing a program to track and log the deleted and zero-length msf file activity, I ran into another issue that may or may not be related, so I wanted to at least mention it.
     3.1.   In Windows 8, 64 bit, I ran into a problem with Explorer and other apps not properly logging file changes. If a file was created, sometimes it would not appear in the log or explorer right away; sometimes if a file was deleted, it would still show up in the App logs or in Explorer. I found a link containing a very lengthy discussion of this problem: "http://social.technet.microsoft.com/Forums/windows/en-US/c8b1f896-bbb7-43b8-be8e-5d28916268e0/windows-explorer-doesnt-refresh-when-movingdeleting-part-3"
     3.2.   I started wondering if it was possible that TB was not seeing file changes either
     3.3.   I was reluctant to mention this issue (3.0) as I know it just adds more complication and does nothing to reduce the complication and find the root cause of the msf file destruction, nevertheless, I thought it couldn't hurt to mention it. 
     3.4.   One of the work-a-rounds listed for this issue was to find and delete any invalid network drive mappings (my interpretation)  -  I did have one, and I did delete it. Now waiting to see what affect it has if any.

I’ll continue testing and logging and check back in a couple weeks if I have anything important more to offer.

Cheers! Craig

Comment 58

[WADA:World Anti-bad-Duping Agency]

A simple/easy workaround of problem of "manual filter run is needed if this bug occurs".
- Define Move To Folder with both "before Classification" and "after Classification".
  Rule set rule-N : consists of 3 rules.
     rule-N-0 : before Classification, if condition-N, do actions except Move to Folder
     rule-N-1 : before Classification, if condition-N, Move to Folder-N
     rule-N-2 : after  Classification, if condition-N, Move to Folder-N
     In any rule, never use action=Delete, action=Stop further filter application.
If "outdatedd msf condition" doesn't exist,
  rule-N-0 is applied, then rule-N-1 is executed normally.
  rule-N-2 won't be executed, because mail is already moved by rule-N-1.
If "outdatedd msf condition" already exists,
  error message is shown(this bug) => you can know folder on which problem occurs
  at this stage, mail is held in Inbox with rule-N-0 applied.
  by rule-N-2, mails are moved from Inbox to Folder-N, without correctly updating .msf
     => "corruption level of Folder-N.msf" is raised.
     This step can be called "Automatic manual filter run feature for this bug" :-)
  You can do manual recovery by "Explicit Folder Open", "Repair Folder" etc.,
  because you already know "folder name on which problem occurred" by fix of bug 782738.

See log data attached to bug 905576 comment #220 ( attachment 8395351 [details] ).

Comment 59

[Lisa Nelson]

I have this same issue, seems to have started just a couple days ago.

ii  thunderbird  1:24.5.0+build1-0ubuntu0.14.04.1  amd64  Email, RSS and newsgroup client with integrated spam filter

Comment 60

[Frederick Sieber]

After removing some sub folders from the "Trash" folder, on the next startup of thunderbird, some filter rules tried to move some messages to Trash (delete message) and this error occurred.  


Log snippet:
2014-06-20 14:38:48.971000 UTC - 0[2b0f140]: nsMsgDatabase::Open(J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Financial.sbd\BanksCreditCards.sbd\Sears Credit Card.msf, FALSE, 2be1120, TRUE)
2014-06-20 14:38:49.031000 UTC - 0[2b0f140]: 6 open DB's
2014-06-20 14:38:49.031000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Inbox.msf - 34 hdrs in use
2014-06-20 14:38:49.031000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Unsent Messages.msf - 0 hdrs in use
2014-06-20 14:38:49.031000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\Auto.sbd\Bernardi.msf - 42 hdrs in use
2014-06-20 14:38:49.031000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\Computer Tracked.sbd\buy.comOld.msf - 361 hdrs in use
2014-06-20 14:38:49.031000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\TrainingTrackReading.sbd\TechTarget.msf - 164 hdrs in use
2014-06-20 14:38:49.031000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\GeneralMerchandise.sbd\TV Deals.msf - 388 hdrs in use
2014-06-20 14:38:49.958000 UTC - 0[2b0f140]: nsMsgDatabase::Open(J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Trash.msf, FALSE, 63a7da0, TRUE)
2014-06-20 14:38:49.958000 UTC - 0[2b0f140]: error opening db 80550005
2014-06-20 14:38:49.958000 UTC - 0[2b0f140]: 7 open DB's
2014-06-20 14:38:49.958000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Inbox.msf - 35 hdrs in use
2014-06-20 14:38:49.958000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Unsent Messages.msf - 0 hdrs in use
2014-06-20 14:38:49.958000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\Auto.sbd\Bernardi.msf - 42 hdrs in use
2014-06-20 14:38:49.958000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\Computer Tracked.sbd\buy.comOld.msf - 362 hdrs in use
2014-06-20 14:38:49.958000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\TrainingTrackReading.sbd\TechTarget.msf - 164 hdrs in use
2014-06-20 14:38:49.958000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\GeneralMerchandise.sbd\TV Deals.msf - 389 hdrs in use
2014-06-20 14:38:49.958000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Financial.sbd\BanksCreditCards.sbd\Sears Credit Card.msf - 344 hdrs in use
2014-06-20 14:38:49.959000 UTC - 0[2b0f140]: closing database    J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Trash.msf
2014-06-20 14:38:49.959000 UTC - 0[2b0f140]: nsMsgDatabase::Open(J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Trash.msf, TRUE, 63a7da0, TRUE)
2014-06-20 14:38:49.959000 UTC - 0[2b0f140]: 7 open DB's
2014-06-20 14:38:49.959000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Inbox.msf - 35 hdrs in use
2014-06-20 14:38:49.959000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Unsent Messages.msf - 0 hdrs in use
2014-06-20 14:38:49.959000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\Auto.sbd\Bernardi.msf - 42 hdrs in use
2014-06-20 14:38:49.959000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\Computer Tracked.sbd\buy.comOld.msf - 362 hdrs in use
2014-06-20 14:38:49.959000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\TrainingTrackReading.sbd\TechTarget.msf - 164 hdrs in use
2014-06-20 14:38:49.959000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\GeneralMerchandise.sbd\TV Deals.msf - 389 hdrs in use
2014-06-20 14:38:49.959000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Financial.sbd\BanksCreditCards.sbd\Sears Credit Card.msf - 344 hdrs in use
2014-06-20 14:38:49.960000 UTC - 0[2b0f140]: closing database    J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Trash.msf
2014-06-20 14:38:49.960000 UTC - 0[2b0f140]: nsMsgDatabase::Open(J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Trash.msf, FALSE, 63a7da0, FALSE)
2014-06-20 14:38:49.961000 UTC - 0[2b0f140]: 7 open DB's
2014-06-20 14:38:49.961000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Inbox.msf - 35 hdrs in use
2014-06-20 14:38:49.961000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Unsent Messages.msf - 0 hdrs in use
2014-06-20 14:38:49.961000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\Auto.sbd\Bernardi.msf - 42 hdrs in use
2014-06-20 14:38:49.961000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\Computer Tracked.sbd\buy.comOld.msf - 362 hdrs in use
2014-06-20 14:38:49.961000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\TrainingTrackReading.sbd\TechTarget.msf - 164 hdrs in use
2014-06-20 14:38:49.961000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\GeneralMerchandise.sbd\TV Deals.msf - 389 hdrs in use
2014-06-20 14:38:49.961000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Financial.sbd\BanksCreditCards.sbd\Sears Credit Card.msf - 344 hdrs in use
2014-06-20 14:38:49.962000 UTC - 0[2b0f140]: closing database    J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Trash.msf
2014-06-20 14:38:56.563000 UTC - 0[2b0f140]: nsMsgDatabase::Open(J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\TrainingTrackReading.sbd\Computer World.msf, FALSE, d0abff0, TRUE)
2014-06-20 14:38:56.761000 UTC - 0[2b0f140]: 7 open DB's
2014-06-20 14:38:56.761000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Inbox.msf - 38 hdrs in use
2014-06-20 14:38:56.761000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Unsent Messages.msf - 0 hdrs in use
2014-06-20 14:38:56.761000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\Auto.sbd\Bernardi.msf - 42 hdrs in use
2014-06-20 14:38:56.761000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\Computer Tracked.sbd\buy.comOld.msf - 391 hdrs in use
2014-06-20 14:38:56.761000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\TrainingTrackReading.sbd\TechTarget.msf - 164 hdrs in use
2014-06-20 14:38:56.761000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Vendors.sbd\GeneralMerchandise.sbd\TV Deals.msf - 388 hdrs in use
2014-06-20 14:38:56.761000 UTC - 0[2b0f140]: J:\Frederick Sieber\AppData\Roaming\Thunderbird\Profiles\mbktn2ov.Default User\Mail\Local Folders\Financial.sbd\BanksCreditCards.sbd\Sears Credit Card.msf - 344 hdrs in use

Comment 61

[Andy Wozniak]

I have observed the same bug as originally described by Craig Lassen. It only seems to happen with some folders while other filters/folders work properly.

Running TB 31.1.2 and Windows 7 Pro.

Comment 62

[bgauthier]

I suffered from this problem for a couple of years. I solved it last week by moving my mail profiles from a Samba-based NAS to the local directory at %AppData%\Thunderbird\. All the symptoms disappeared with this change: slow processing, slow compacting, missed message filtering. It is possible that the latter were a side effect of slow compacting (or other) operations.

Note that the problem is not with the NAS (which works fine with other applications, including heavy data base applications).

Note also that slowness associated with other applications (in particular, Powerpoint) reading from and writing to the NAS also disappeared with my moving Thunderbird profiles off the NAS.

Finally, I should note that my profiles include relatively small message files/folders. I keep them very tidy, with fewer than 500 messages in the largest folders and less than 20 megs in size.

Comment 63

[Craig Lassen]

TB Programmers,

I wanted to give you some feed-back and let you know that the original problem is much, much, much, better than it use to be, I'd say either completely bearable or fixed now.

Ever since one of the recent TB updates, maybe a couple updates ago or so, the problem almost completely resolved itself. Very occasionally now, I will get an error on a folder. But only when returning after at long period of time (at least four day) and downloading hundreds (or thousands) of emails at a time.

When it happens, which is rare, it never causes the kind of corruption that it use to cause to other folders in the same sub-directory. Now, when it happens, I either ignore it, or sometimes run the message filter manually on the inbox and the problem is never seen again.

Thank you so much for the code changes that helped to mitigate this problem. Your dedication and nothing short of absolutely amazing!

I'd like to especially thank WADA, as I believe it was his in-depth analysis of this problem that allowed this issues to be isolated enough to where an effective rewrite could be crafted.

Craig

Comment 64

[Andy Wozniak]

(In reply to Andy Wozniak from comment #61)
> I have observed the same bug as originally described by Craig Lassen. It
> only seems to happen with some folders while other filters/folders work
> properly.
> 
> Running TB 31.1.2 and Windows 7 Pro.

Just a bit more info. My windows PC is standalone and does not access a samba server. The problem appears only with my gmail POP account. Filters for verizon POP accounts work properly with errors. Both use SSL/TSL security.

Comment 65

[Andy]

I experience this error a lot. 

Since it goes away after disabling message filters it looks like a locking problem. I use several accounts and when emails have to be moved from several accounts to a single one at the same time this seems to happen. 

Instead of one thread locking a folder for writing and the other waiting for it to be released, they both try to access it at the same time and the software crashes.

Could that be the case?

Comment 67

[Tim Johnson]

System info: TB 38.7.0. Windows 10 Pro 64 bit, latest updates applied.

I started to see this issue, first without any error messages, a few weeks ago when I started changing e-mail addresses of various subscriptions from my cox.net account to one with a domain I own. For convenience, I created a filter on the inbox of the second address to move selected mail to the input of the cox.net domain address.

What I observed is mail to the second address which was downloaded upon resumption from hibernation, would fail to be moved to the cox.net inbox; no error message apparent. Mail to the second address arriving somewhat after resumption from hibernation was always correctly filtered. Manually running the filter on the second address inbox would correctly move the originally unmoved mail to the cox.net inbox.

In trying to determine why the filter failure to move upon resumption from hibernation, I stopped TB before hibernation in the evening, and started a new instance of TB the next morning after resumption from hibernation. This time, I got a pop-up stating "The messages could not be filtered to folder 'Inbox' because another operation is in progress."

This led me to this bug. I believe, as others here have pointed out, this is due to a TB design or implementation decision to not use appropriate synchronization techniques to avoid a thread safety issue. Using mutexes or similar means to synchronize between contending threads would avoid this issue.

I have a work-around where I created a new folder for the cox.net address and changed the filter to move to that folder instead of the cox.net inbox folder. This seems to have eliminated the contention for the inbox resource.

Comment 68

[marty]

I get an Error at start-up of Thunderbird when it automatically starts filtering incoming new messages to IMAP subscribed folders. 

The Filter log entry of the failure is:
"Filter Action Failed: "Move failed" with error code=0x8055000f"

When I manually run the filters on the in-box folder, then it works fine.

My configuration is: 
Win7x64 version: 6.1.7601
Thunderbird version: 38.7.2

Comment 69

[Christov.bio]

I fixed this problem thanks to the posting in comment 39 from Graig and also thanks to this blog article which pointed me to the right action:
http://qdosmsq.dunbar-it.co.uk/blog/2014/06/thunderbird-message-filters-stop-working-on-email-download-but-work-ok-manually/
Thank you!

Comment 70

[jayhanson]

Christov's work-around (shown above) solved my problem too. However, I am adding an observation for debuggers:

When I expanded some of my long-unused subfolders, I noticed that although some of the message headers were bolded (shown as being unread), the folder view did not indicate any messages were unread. I am hoping the mismatch between the folder view and the header view might provide a hint of the real problem which still underlies this bug.

As I said above, this procedure http://qdosmsq.dunbar-it.co.uk/blog/2014/06/thunderbird-message-filters-stop-working-on-email-download-but-work-ok-manually/ made the symptom go away -- for now.

Thanks to Christov,
Jay

Comment 71

[WADA:World Anti-bad-Duping Agency]

About following in History.
> [e-mail] 2013-11-15 02:48:45 PST Keywords regression
> Whiteboard [regression:??]

If definition of "regression" is "new problem what was not observed in any former builds", this bug is "regression".
But in my experience in the past, "regression" was used for "problem of already resolved/fixed bug came back" in parent company(grandfather or great‐grandfather may be correct, HQ is located in east side of US, never west side such as California :-) ) of my company in Japan where I was hired.
If follow my parent company's definition, this bug is never "regression".
This bug is rather an "improvement" by new error detection and new error message which was never shown by Thunderbird due to no error check in code.

Dark side of this bug is :
 Message is not moved by filter when the error message is shown,
 although message was moved by former releases.

However, in former releases and in "Filter after Junk Classification", message data was appended to msgStore file but msgDBHdr in msgDatabase was/is not updated.
So, "outdated msf condition" was/is kept after filter execution.
This produces phenomenon like "message count at folder pane is not updated".
Then, upon next explicit folder open(eg. folder click at folder pane), Rebuild-Index(Repair Folder) was/is invoked internally.

I believe "new error detection" is "improvement" and is never "regression".

Comment 72

[Wayne Mery (:wsmwk)]

> If definition of "regression" is [1] "new problem what was not observed in any former builds", this bug is "regression".
> But in my experience in the past, "regression" was used for [2] "problem of already resolved/fixed bug came back" in parent company

For our purposes (produce quality purposes), regression is both of these. :) So it doesn't matter whether there the problem was only caused by a known patch, [2] above.

Note - the "??" in whiteboard is a placeholder for a version that has not been identified. It is not questioning whether there is a regression.

Comment 73

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #72)
> Note - the "??" in whiteboard is a placeholder for a version that has not been identified.
> It is not questioning whether there is a regression.

Roger, but it's confusing...

Comment 74

[jayhanson]

With respect to Christov's work-around (shown above). When I expanded some of my long-unused subfolders, I noticed that although some of the message headers were bolded (shown as being unread), the folder view did not indicate any messages were unread.

I offer one final question and observation for the debuggers. How did my folders get screwed-up in the first place?

Although I can't be certain, the problem very well could have originated months (or even years) ago when I moved many thousands of messages from one folder to another, with Tbird terminating before the move completed. This occurred several times. Sometimes I canceled it with task manager. Sometimes it just seemed to get screwed up.  It's something that could be tested.

I hope not to be back here,
Jay

Comment 75

[mark]

Note: after bug presents itself and mail is not moved to the expected filtered folders and all mail has been received, to get the mail into the proper folders, you just have to use Tools/Run Filters On [incoming] Folder and all of the unmoved mail will then move to the expected filtered folders. I add this since it appears that nobody reported it.

Comment 76

[Dan Howard]

I get a similar message:

The messages could not be filtered to folder 'Trash' because another operation is in progress.

It happens if I have 2 filters pointing to local Trash. One being regular email and one being gmail.   You should just add a wait on that thread instead of erroring out.

Comment 77

[Michelle]

So I was misinformed by the forum I always go to when I have issues. They said that TB can only filter one rule at a time.

This only happens when I have incoming mail. If I go to Tools - Run Filters on Folder, all is fine.

Here's the error I get & it's very frustrating b/c if I have a lot of mail that comes in, it will stop everything until I press "ok" for this popup.

https://www.screencast.com/t/5eFav7R6rG

TB vs 52.6.0

Thanks

Comment 78

[Michael Tokarev]

We're seeing this issue quite frequently here. It usually happens when a batch of new emails is coming when several of them are being moved to the same folder, but sometimes it is single email which causes this. It happens several times a day for one account, with multiple accounts having this property. All email folders are on a samba network share, thunderbird works on windows10 machines (multiple). Current version of TB, the issue is here for quite some time. Just repeating the filter manually ("Run message filters on this folder") makes it to go, until the next hiccup. Rebuilding (with or without removing) the .msf files does not help.

Comment 79

[Michael Tokarev]

The error message in filterlog.html mentions error=0x8055000f each time this issue occur.

Comment 80

[Wayne Mery (:wsmwk)]

There are multiple indications of how filtering can fail:

• no error message - bug 1534153
• The messages could not be filtered to folder '%S' because adding a message to it failed... - Bug 1973733
• The messages could not be filtered to folder '%S' because writing to folder failed - this bug