905576 - TB3 repeatedly rebuilding summary files for many folders when profile(mail directory) is on network drive. (Fileshare server is SAMBA on Linux, Fileshare client is Windows)

Status: NEW

(MailNews Core :: Database, defect)

Description

[Matthew Jurgens]

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0 (Beta/Release)
Build ID: 20130730113002

Steps to reproduce:

Thunderbird data files on a Samba network share. Folder rebuilds occur often.

Please refer to https://getsatisfaction.com/mozilla_messaging/topics/tb3_repeatedly_rebuilding_summary_files_for_many_folders?utm_content=topic_link&utm_medium=email&utm_source=reply_notification


Actual results:

Folder rebuilds occur

Please refer to https://getsatisfaction.com/mozilla_messaging/topics/tb3_repeatedly_rebuilding_summary_files_for_many_folders?utm_content=topic_link&utm_medium=email&utm_source=reply_notification


Expected results:

Rebuilds should not be occuring.

Also refer to this existing bug
https://bugzilla.mozilla.org/show_bug.cgi?id=624806#c16

Comment 1

[WADA:World Anti-bad-Duping Agency]

If Thunderbird is not normally terminated and mail folder is not correctly closed, it's pretty normal phenomenon. For example,
- Tb is killed by Task Manager.
- Termination of Tb is requested => Tb's window is closed.
  After Tb's window is closed, power off PC immdiately,
  while Tb's termination process is still running.
- mail folder file is held in USB memory.
  Terminate Tb => Tb's window is closed, but Tb's trmination process
  is still in progress.
  Before end of Tb's termination, pull off USB memory from PC.
  Note: This is actually reported daily operation by a user who used
  Portable Thunderbird, although reported problem was "address book
  is not correctly saved".
Was your Thundebird normally terminated? Did termination process of Thunderbird always end normally and completely?

SMB2? or SMB1? Problem due to SMB2 like bug 545650?
If SMB2, is disabling SMB2 a workaround?

Problem on msf. file for local mail folder? (POP3, or Local Folders)
Or problem on .msf file for IMAP mail folder?

When local mail folder, for mail folder named "Mbox", final filesize and timestamp(last modified) of file named "Mbox" is written to file named "Mbox.msf" upon close of mail folder named "Mbox".
If the "file size/timestamp saved in Mbox.msf file" is different from "file size/timestamp of file named Mbox.msf" which is passed from OS upon next mail folder open, "outdated msf" condition occurs(.msf file was no properly closedd by previous folder access). If the "outdated msf" happens, Internal rebuild-index(Repair Folder) is invoked. This is design since Netscape Messenger.

If relocation of files is executed at file server, and if file size/timestamp is altered by the "relocation of files", above "outdated msf" occurs.

Does your file server alter file size and/or timestamp of files held at server?

As written in bug 232047 comment #12, difference of "file timestamp" among OSes is known. And the difference can produce "outdated msf" condition when mail related filess are held at file server or Tb's mail directory is shard by multiple Tbs on different OSes. So, mail.db_timestamp_leeway(deault=4000, perhaps 4000 seconds) was introduced and currently defaulted to "4000".

Is larger mail.db_timestamp_leeway value a workaround in your case?

Anyway, "outdated msf" conditon can be logged by following.
  SET NSPR_LOG_MODULES=timestamp,MSGDB:5
(error upon Mbox open is logged if outdated-msf condition exists)
Get log, and check log file content by Text Editor.
See bug 402793 comment #28 for getting NSPR log.

Before getting log in your daily use, get log with minimum configuration, with minimum activity, and view log file and check log file content by yourself first, please.

Comment 2

[Matthew Jurgens]

No abnormal shutdowns.
SMB1.
I only use POP and local folders.
Server does not relocate files and only changes modified timestamp upon client write request eg Thunderbird.
Leeway=4000, larger value may be a workaround.
I have logging started, now what do I look for to find folder rebuilds?

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #2)
> now what do I look for to find folder rebuilds?

You said "Folder rebuilds occur often", so I assumed "Rebuild index(==Repair Folder)" actually occured often. How can I know when "Folder rebuilds" you call will occur?

What kind of phenomenon do you call "Folder rebuilds occur often"?
Indexing by "Global Search and Indexer" instead of "rebuild index"?

By the way, "outdated msf" condition is pretty easily emulated.
(1) Create a new local mail folder(Call Test/FolderX, FolderX under Test), and copy a simple text mail to Test/FolderX.
(2) Terminate Tb.
(3) Edit file named ...\Test.sbd\FolderX by Text Editor, duplicate all mail data lines which starts with "From - <timestamp in local time>",
and change Subject and Message-ID of second(duplicated) mail.
  This is emulation of;
    Second mail is appended to FolderX when first mail exists,
    and file of FolderX is normally closed.
    Data of Folderx.msf is still held in memory,
    becausee mail folder named FolderX is not closed yet.
    Power failure of PC happens.
    => mismatch between "file meta data of FolderX" and "FolderX.msf
       content" happens.
(4) Restart Tb, with logging enabled.
(5) Click FolderX under Test at folder pane
    => Rebuild index is invoked, and second(duplicated) mail is shown.
(6) Terminate Tb, and check log file content.
    You can perhaps see log for "folder open error".
See what kind log is generated by "outdated msf" condition first, please.

Comment 4

[Matthew Jurgens]

Attachment: Video showing how Thunderbird rebuilds folders

Comment 5

[Matthew Jurgens]

Attachment: Log to match attached video

This is the log file that was captured while the attached video was made. This log has been created using 
SET NSPR_LOG_MODULES=timestamp,MSGDB:5
set NSPR_LOG_FILE=%USERPROFILE%\Desktop\TB.log
"c:\program files\Mozilla Thunderbird\thunderbird.exe"

This log contains multiple folder rebuilds

Comment 6

[Matthew Jurgens]

I have just uploaded a video showing how the folder rebuilds occur and a matching log file. More than one folder rebuilt during this captured run including the Trash folder. The leeway setting was 4000.
The captured run was only after about 1 minute from the previous run of Thunderbird which also touch the Trash folder. 
The clock of the server and the client PC are time synced to each other and would only be a couple of seconds out if that.

Comment 7

[Matthew Jurgens]

I should also say that I am currently running TB 24 Beta

Comment 8

[WADA:World Anti-bad-Duping Agency]

> Bug summary : (snip) when profile is on network drive.

It looks both "profile on network drive(with mail directory under profile)" and "mail directory on network drive which is different from profile directory" are involved in your case.

(i) profile on network drive(with mail directory under profile)
> S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\smart mailboxes\Inbox.msf
> S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\pop3.edcint.co.nz\Inbox.msf
(ii) mail directory on network drive which is different from profile directory.
> S:\data\Mozilla Thunderbird\Matthew\edcint.co.nz\Inbox.msf
> S:\data\Mozilla Thunderbird\Matthew\Local Folders\Inbox.msf

How do you mount the "network drive" on MS Win?
Is same drive letter always assigned to the network drive?
Or you use UNC path for profile directory(path in profiles.ini) or mail directory of Tb?

(a) File path saved in panacea.dat(Folder Cache) is absolute path,
    instead of "relative path from profile directory".
    This is similar to "absolute path in extensions.ini",
    and this can be called "current restriction".
(b) MS Win uses "Drive letter".
Different drive letter assignment means; 
(c-1) "profile directory" is moved to different location.
(c-2) "mail directory" is moved to different location.
      mail directory : 
        Directory specified at "Local Directory:" of Server Settings
        which is saved in mail.server.serverN.directory-rel of prefs.js.
And, (c-2) may force "rebuild index", because "path of mail directory" is different from "path saved in Folder Cache".

Comment 9

[WADA:World Anti-bad-Duping Agency]

"Folders for which open error is reported" are following.
"Building summary ..." is for these folders, isn't it?
What is you purpose to place files for mail folder in %temp% directory?

> 2013-08-19 10:35:11.223000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Sent.msf, FALSE, 5360b30, TRUE)
> 2013-08-19 10:35:11.223000 UTC - 0[e0f140]: error opening db 80550006

> 2013-08-19 10:35:24.676000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\eBay.msf, FALSE, a6dd410, TRUE)
> 2013-08-19 10:35:24.676000 UTC - 0[e0f140]: error opening db 80550006

> 2013-08-19 10:35:25.285000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Nagios-Users.msf, FALSE, 5361250, TRUE)
> 2013-08-19 10:35:25.285000 UTC - 0[e0f140]: error opening db 80550006

> 2013-08-19 10:35:26.192000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Trash.msf, FALSE, 9d93110, TRUE)
> 2013-08-19 10:35:26.192000 UTC - 0[e0f140]: error opening db 80550006

> 2013-08-19 10:35:30.463000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Sent.msf, FALSE, 9d94540, TRUE)
> 2013-08-19 10:35:30.463000 UTC - 0[e0f140]: error opening db 80550006

Comment 10

[Matthew Jurgens]

Attachment: panacea.dat

Comment 11

[Matthew Jurgens]

Attachment: prefs.js

Comment 12

[Matthew Jurgens]

The folders and profile are mounted consistently from the same place. The profiles.ini file contains:
IsRelative=0
Path=S:\data\Mozilla Thunderbird\Profiles\Matthew

The drive letter is consistent and has been the same path as used for Thunderbird from and including the 2.x series (where there was never a problem). In fact I have purposely keep a 2.x series client that still, to this day never experiences a problem using the same drive mapping and location.

I've also just uploaded prefs.js and pancea.dat

I'm wondering if there is a problem with the theory 'because "path of mail directory" is different from "path saved in Folder Cache".' in that the folder rebuilds do not consistently happen and the configuration never changes. I have in fact never been able to finds a pattern to the rebuilds and make them happen on demand. I did actually have to try multiple times to get that attached video and log with the fault because it was just not happening.

Comment 13

[WADA:World Anti-bad-Duping Agency]

As seen in following line, NS_MSG_ERROR_FOLDER_SUMMARY_OUT_OF_DATE is 0x80550005.
> http://mxr.mozilla.org/comm-central/source/mailnews/db/gloda/modules/index_msg.js#41
This error code is defined by;
> http://mxr.mozilla.org/comm-central/source/mailnews/base/public/msgCore.h#104
> 104 #define NS_MSG_ERROR_FOLDER_SUMMARY_OUT_OF_DATE NS_MSG_GENERATE_FAILURE(5)
> 105 #define NS_MSG_ERROR_FOLDER_SUMMARY_MISSING NS_MSG_GENERATE_FAILURE(6)

So, following error is : NS_MSG_ERROR_FOLDER_SUMMARY_MISSING
> nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Sent.msf, FALSE, 5360b30, TRUE)
> error opening db 80550006
i.e. C:\ ... \Temp\MozillaMailnews\Sent.msf is not found for Sent folder in mail directory which is defined under %Temp% directory.
i.e. Rebuild Index will be forced for this folder defined in %Temp% directory which is never mail folder on network drive.

Funny mail directory setting in your prefs.js is following only.
> user_pref("mail.server.server3.directory", "C:\\Documents and Settings\\mjurgens\\Application Data\\Thunderbird\\Profiles\\p2951ou3.default\\Mail\\smart mailboxes");
> user_pref("mail.server.server3.directory-rel", "[ProfD]Mail/smart mailboxes");

Even though profile directory is directory on S: drive, why relative path of "[ProfD]Mail/smart mailboxes" is converted to absolute path of 
"C:\\Documents ... \\Profiles\\p2951ou3.default\\Mail\\smart mailboxes".

Because following is seen in panacea.dat, Tb correctly accesses Virtual Folders defined in S:\\data\\ ... \\smart mailboxes which is pointed by Path=S:\data\Mozilla Thunderbird\Profiles\Matthew in profiles.ini and mail.server.server3.directory-rel=[ProfD]Mail/smart mailboxes.
>     =S:\\data\\Mozilla Thunderbird\\Profiles\\Matthew\\Mail\\smart mailboxes\\\
> Trash.msf)
And, because "C:\\ ... \\smart mailboxes" is not seen in panacea.dat, Tb doesn't look to access "C:\\ ... \\smart mailboxes".

It's simply that mail.server.serverN.directory is not updated if hidden "smart mailboxes" account?

Another concern : ":" in userName then "%3A" in mailbox: URL.
> user_pref("mail.server.server8.userName", "mail.ezyfoods.com:[e-mail]");
> user_pref("mail.server.server8.spamActionTargetAccount", "mailbox://mail.ezyfoods.com%3Amatthew.jurgens%40ezyfoods.com@gw");
> user_pref("xpunge.options.multi.compact.accounts", "mailbox://mail.edcint.co.nz%3Amjurgens%40edcint.co.nz@gw   mailbox://mail.edcint.co.nz%3Adefault.mailbox%40edcint.co.nz@gw   mailbox://mail.smartmon.com.au%3Amjurgens%40smartmon.com.au@gw   mailbox://mail.ezyfoods.com%3Amatthew.jurgens%40ezyfoods.com@gw   mailbox://nobody@Local%20Folders");
> user_pref("mail.last_msg_movecopy_target_uri", "mailbox://mail.edcint.co.nz%3Amjurgens%40edcint.co.nz@gw/Bulk/Nagios-Users");
(Nagios-Users is file name on which "error opening db 80550006" occurs)

This may cause mailbox: URL parsing error which is used in virtual folder definition of "smart mailboxes" account for "Unified Folder". It may cause "%Temp% directory access by Tb in searching mail box files.

Other concern : add-on
Add-on may use Berkley Message Database internally with using %Temp% directory.
Does your problem occur with -safe-mode of Tb? ("...\thunderbird.exe" -safe-mode)

Comment 14

[Matthew Jurgens]

1. The username in the format HOST:EMAIL is ok. I use that to forward through a POP3 Proxy to pickup SPAM
2. I have reproduced the problem in safe mode (and can confirm the add-ons were disabled). I tried doing logging but logging in the same manner with environment variables did not seem to produce any files.

Comment 15

[WADA:World Anti-bad-Duping Agency]

> 2013-08-19 10:35:25.285000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Nagios-Users.msf, FALSE, 5361250, TRUE)
> 2013-08-19 10:35:25.285000 UTC - 0[e0f140]: error opening db 80550006
"C:\DOCUME~1\　... \Temp\MozillaMailnews\Nagios-Users.msf" not found occurs.

> user_pref("mail.last_msg_movecopy_target_uri", "mailbox://mail.edcint.co.nz%3Amjurgens%40edcint.co.nz@gw/Bulk/Nagios-Users");
Folder named "Nagios-Users" is owned by server8.

> user_pref("mail.server.server8.directory", "S:\\data\\Mozilla Thunderbird\\Matthew\\ezyfoods.com");
> user_pref("mail.server.server8.directory-rel", "[ProfD]../../Matthew/ezyfoods.com");
Mail directory for server8 is "S:\\data\\Mozilla Thunderbird\\Matthew\\ezyfoods.com".

If "S:\\data\\Mozilla Thunderbird\\Matthew\\ezyfoods.com\\Bulk" is symlink or mounted folder for "C:\DOCUME~1\　... \Temp\MozillaMailnews", acessing to "C:\DOCUME~1\　... \Temp\MozillaMailnews\Nagios-Users may occur, and Tb may try to recreate "Nagios-Users.msf".

Do you use symlink(or Mounted Folder)?

Comment 16

[Matthew Jurgens]

I can not find (and do not want) and mail files on my C:
I'm therefore hoping that these errors are meaningless and irrelevant.

There are no symlinks in or below S:\\data\\Mozilla Thunderbird

s: (The CIFS share)is mounted as //gw/shared ie way above the Mozilla folder.

Perhaps the server8/Nagios Users folder is red herring.

What if we focus on a specific folder. Folders that are frequently used are more likely to get rebuilds. I just had a rebuild on the SPAM folder. Possibly making this a simple case is the following:
1) There is only one of these across all my mail accounts
2) it does not show up as a unified folder.
3) it is not in prefs.js
4) It is not directly associated with any POP/IMAP accounts - it is a local folder
5) it only gets mail put in there from rules from other accounts (to filter mail server reported spam)
6) There are only 3 files I can find that relate to this folder: 
./Matthew/Local Folders/Spam.msf
./Matthew/Local Folders/Spam
./Matthew/Local Folders/Spam.sbd

Comment 17

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #16)
> What if we focus on a specific folder. Folders that are frequently used are more likely to get rebuilds. 
> I just had a rebuild on the SPAM folder.
> (snip)
> 5) it only gets mail put in there from rules from other accounts (to filter mail server reported spam)
> 6) There are only 3 files I can find that relate to this folder: 
> ./Matthew/Local Folders/Spam.msf
> ./Matthew/Local Folders/Spam

When mails are moved to Local Folders->Spam by message filter upon mail download from server, "Unread mail count of the Local Folders->Spam" is automatically updated(incremented, and message filter log for "move to Local Folders->Spam" is logged.

(Q1) Was "Unread mail count of the Local Folders->Spam" updated correctly always by the filter move?

If following occurs upon filter move to Local Folders->Spam,
> File size of Spam.msf == 0,
> NS_MSG_ERROR_FOLDER_SUMMARY_OUT_OF_DATE
> NS_MSG_ERROR_FOLDER_SUMMARY_MISSING
phenomenon of bug 495760 or phenomeno of bug 261419 occurs.
So, following occurs.
- mail data is successfully appended to file named Spam.
- MsgDataBase information(mail meta data etc.) is not correctly
  written back to Spam.msf.
- So, Unread mail count is not updated at folder pane.
- Filter log for "move to Spam" is written.
- All of above occurs upon each mail download and filter move to Spam
  by filter, until explicit folder open is executed by user.
- Upon folder click of Local Folders->Spam at folder pane,
  (explicit Spam folder open),
  Rebuild Index(rebuild summary file, Repair Folder) is invoked,
  and unread mail count is updated at folder pane,
  and newly moved mails by filter are shown at thread pane.

(Q2) When did you saw the "rebuild on the SPAM folder"?
By folder click? upon restart of Tb? Or when SPAM folder of Local folders is opened at folder pane and mails are shown at thread pane? Or other timing?

TCP Window Scaling is enabled on Win PC by default and it's perhaps same on your SAMBA server.
> http://en.wikipedia.org/wiki/TCP_window_scale_option
As well known, Windows Server 2003 had many poblems in file share with SMB after enabling SNP feature which contains TCP Window Scaling support. And, solution by MS for such problems in Windows Server 2003 was to provide "Update to turn off SNP features for Windows Server 2003".
> http://support.microsoft.com/kb/948496

If similar issue to bug 545650 by SMB2 occurs with SMB1, Spam.msf corruption by Tb may occur.

(Q3) Is similar issue exists in SAMBA? Is latest version is used at your SAMBA server(OS, SAMBA, ...)

Comment 18

[WADA:World Anti-bad-Duping Agency]

In the SMB protocol, opportunistic locking is a file locking mechanism.
> http://en.wikipedia.org/wiki/Server_Message_Block
Because of "opportunistic locking", file access at different PC/Server can interfare(by change Level 2 OpLocks, Breaks) Tb's file access via File Share.

(Q4) No fSpam.msf file access at different PC\Server while Tb is using Spam.msf?

Comment 19

[Matthew Jurgens]

Q1: I can not tell you the answer to this question since the filter rule marks the mail as read. From when this happens to other folders the the unread count does not properly update.

Q2: The rebuild of the SPAM folder (this is not a Mozilla Thunderbird Junk folder, this is a folder I call spam that receives pre-filtered spam from my mail server) happens normally only when I use a virtual folder powered by a search rule (as described in my original report https://getsatisfaction.com/mozilla_messaging/topics/tb3_repeatedly_rebuilding_summary_files_for_many_folders - see the paragraph starting with "If you happen to use the virtual folders"). So in this case the spam folder is accessed by the search rule trying to build the virtual folder. I have found this the easiest way to touch all my folders so I do not manually have to try and find the ones that needed rebuilding.


Q3: I am running Fedora 17 with Samba 3.6.12. This is currently the latest Samba for Fedora 17. Since initially reporting this over 3 years ago I was probably on Fedora 10 or something like that.

TCP Window scaling is enabled in my kernel 
cat /proc/sys/net/ipv4/tcp_window_scaling
1
cat /proc/sys/net/ipv4/tcp_rmem 
4096    87380   6291456

Q4: I have no oplock setting in the smb.conf file. I think this means that it is turned on by default. I can disable it if needed. There are 2 other ways that these files would be used
1) By another instance of Thunderbird running using the same files. Only I do this and this is currently not happening. Thunderbird won't run any way as it find the parent lock file or something in used and refuses to start on the 2nd instance.
2) By my backup process which runs at scheduled times (using rsync). It should opening the files readonly. I know when these backups run and outside the backup times I see folder rebuiulds randomly.

Do you want more debug logging turned on?

Comment 20

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #19)
> Q2: (snip) 
> So in this case the spam folder is accessed by the search rule trying to build the virtual folder.
> I have found this the easiest way to touch all my folders so I do
> not manually have to try and find the ones that needed rebuilding.

It was only way to force "rebuild-index of all folders/subfolders for which xxx.msf was manually/intentionally deleted" without manually touching all mail folders/subfolders at folder pane, which I could find while testing "manual deleted .msf file" case.
  Filter copy/move => rebuild-index was not invoked by Tb.
  So I created Search Folder and opened the Search Folder
  -> rebuild-index was invoked
If I undestand correctly, "Access to search target folder by Search(Virtual) Folder open" is identical to "Explicit open of the search target folder".

> Do you want more debug logging turned on?

No.
Because "filter move to Spam" is logged in message filter log(filterlog.html under "Local Folders" directory) and "download timestamp" is written in "From - timestamp" line in .../Local Folders/Spam file(Local timestamp without timezone indicator), "When mails are downloaded and moved to Spam folder" can be traced by message filter log and "From - ..." line.
Please check timestamp relevant to "download from server", "move to Spam" when you will be aware of "rebuild-index of Spam folder".

> 2) By my backup process which runs at scheduled times (using rsync). It
> should opening the files readonly. I know when these backups run and outside
> the backup times I see folder rebuiulds randomly.

Even if "Read only access" and even if "Read open at different PC", if it's done just before "Read/Write Shared open of Spam.msf file by Tb", it produces "error return code to open Spam.msf by filter move of Tb", and filter move code doesn't check error return code well(see bug 695671 comment #37). So, "open error" may produce status like "file size of Spam.msf==0" or "End of file pointer==0" in Tb.
See bug 498814 and bug 674742 for similar issue in Compact code, please.

> Q4: I have no oplock setting in the smb.conf file. I think this means that it is turned on by default.

Oplock is used by both SMB1 server and SMB1 client.
  SMB1 server : latest SAMBA on recent Linux
  SMB1 client : File share of MS Windows XP(perhaps SP-3)
Similar phenomenon was reported to forum in Japan.
  SMB1 server : Windows Server 2003
  SMB1 client : File share of MS Windows 7
I'm suspecting Oplock/Cache relevant problem while using SMB1 when Tb uses not-so simple and popular file access technique(not simple "open and read until end of file", not simple "open and write entire data").

If SMB1 client requets Batch lock for "Spam.msf open by Tb",
> http://en.wikipedia.org/wiki/Server_Message_Block#Batch_Locks
Following may occur.
1. Tb writes data to Spam by filter move and close Spam after filter
   move.
   Because Tb has problem of bug 769346, this step takes suficiently
   long if network file is used.
Note:
This step is always executed successfully, because phenoenon like next is never observed on folder where problm of "rebuild-index is invoked" occurs.
- mail is not moved by filter even though "move to folder" is logged
  in filter log. (folder lock failure, open failur of .msf, etc.)
- mail data is truncated at move target folder.
  (I/O error while writing data to Spam, before close Spam)
2. Tb write back Spam.msf content to Spam.msf file.
   Because Tb has problem of bug 769346, this step also takes long
   if network file is used.
3. Next download occurs and filter move occurs.
4. "Close Spam.msf" and "Open Spam.msf" is requested consecutively.
5. Because of Batch lock, two SMB1 requests for close/open cancel
   each other.
6. When step 5 occurs, "End of file position=0" is returned to Tb
   after open of Spam.msf file.
   (similar issue to bug 545650 caused by bug of SMB2)
   => Tb considers file size of Spam.msf==0.

Can you check whether such problem happens in your environment?

Comment 21

[Matthew Jurgens]

I am now having trouble reproducing it with the SPAM folder. I have packet matching video, packet captures and TB logs for Trash and Sent rebuilds. Do you want that instead?

Comment 22

[Matthew Jurgens]

For new anyway, I have uploaded all the debug files I have to:
http://www.edcint.co.nz/LargeFiles/Completed/

Comment 23

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #21)
> I am now having trouble reproducing it with the SPAM folder.

"Which folder" is not important. "reason code of error opening db" which produced rebuild-index is important.

> Do you want that instead?

No. Data when rebuild-index is invoked and executed is usually useless, unless cause of "error opening db" was ".msf file open error" like event which occurred on the .msf file first time just when Tb tries to open the mail folder. Data what is needed for problem analysis by developers is log/trace/dump etc. just when cause of "error opening db" was generated.

(In reply to Matthew Jurgens from comment #22)
> For new anyway, I have uploaded all the debug files I have to:
> http://www.edcint.co.nz/LargeFiles/Completed/

"error opening db" in Tb log.
> http://www.edcint.co.nz/LargeFiles/Uploading/3.TB.log/split_aaaaaa
> 2013-08-26 22:03:57.101000 UTC - 0[f0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Sent.msf, FALSE, fe14b0, TRUE)
> 2013-08-26 22:03:57.101000 UTC - 0[f0f140]: error opening db 80550006
> http://www.edcint.co.nz/LargeFiles/Uploading/3.TB.log/split_aaaaac
> 2013-08-26 22:04:18.226000 UTC - 0[f0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Trash.msf, FALSE, fe1710, TRUE)
2013-08-26 22:04:18.226000 UTC - 0[f0f140]: error opening db 80550006
As I already wrote, 80550006 is NS_MSG_ERROR_FOLDER_SUMMARY_MISSING.

Is these log written when you opened virtual folder?

Obviously, this is never "file on network drive".
It may be "temporary database for virtual folder". If so, it may be interfere by other software.
- Tb write data to Sent(Trash).msf and close Sent(Trash).msf.
- Just before Tb tries to open Sent(Trash).msf for folder processing,
  someone opened Sent(Trash).msf.
- Because "open read/write shared mode" fails,
  Tb considers it ".msf file is missing".

Comment 24

[Matthew Jurgens]

I have uploaded some more files. They all start with 5.
There are 4 TB opens recorded.
The first 2 times the trash folder got rebuilt and the first 2 logs appear to have matching lines:
"nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Trash.msf"
There is a packet capture containing all SMB traffic.

Comment 25

[WADA:World Anti-bad-Duping Agency]

FYI.

"MozillaMailnews" was a special literal which is hard coded in source.
> http://mxr.mozilla.org/comm-central/source/mailnews/base/util/nsMsgDBFolder.cpp#3880
> 3880   rv = path->Append(NS_LITERAL_STRING("MozillaMailnews"));
This is a line in followig method.
> 3869 /* Finds the backup directory associated with this folder, stored on the temp
3> 870    drive. If that path doesn't currently exist then it will create it. Path is
> 3871    strictly an out parameter.
> 3872   */
> 3873 nsresult nsMsgDBFolder::CreateBackupDirectory(nsIFile **resultFile)

Bugs which contains "backup database" in bug sumary?
> https://bugzilla.mozilla.org/buglist.cgi?list_id=7764125&short_desc=%20backup%20database&chfieldto=Now&query_format=advanced&chfieldfrom=3y&short_desc_type=allwordssubstr
Bug 438369, Bug 476911 looks relivan to to "BackupDatabase".

And, you uae Lightning...
> user_pref("calendar.preferences.lightning.selectedTabIndex", 4);

Do you know about backup directory or backup database in Tb?

Comment 26

[WADA:World Anti-bad-Duping Agency]

What kind of data is held in C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Trash(Inbox, Sent etc.) of client PC(MS Win)? Which account's mail data? What is file size and timestamp? When was mails downloaded?(see "From - ..." line)

Comment 27

[Matthew Jurgens]

At the moment: C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews is an empty directory.
Yes I use lightning.
I do not know about the backup directory or database.

Comment 28

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #27)
> At the moment: C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews is an empty directory.

It sounds that C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Trash(Inbox, Sent etc. is deleted by Tb after "error opening db 80550006" occurs on C:\DOCUME~1\ ... \???.msf.

Can you check file access log on C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Trash(and Trash.msf) by Process Monitor?
> http://technet.microsoft.com/en-us/sysinternals/bb896645
By "SET NSPR_LOG_MODULES=timestamp,MSGDB:5,MsgCopyService:5", Tb log for MsgCopyService is also obtained. If MsgCopyService is used for C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Trash(Inbox, Sent etc.) access by Tb, Tb log for it may help our problem analysis, because part of C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Trash(and Trash.msf) folder access by Tb is logged.

Another question.
What will happen if C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews is deleted?

Comment 29

[WADA:World Anti-bad-Duping Agency]

Temporary database was used by "Repair Folder" himself, and Temp\MozillaMailnews is always created if it doesn't exist.

GetBackupSummaryFile is called here.
> http://mxr.mozilla.org/comm-central/source/mailnews/base/util/nsMsgDBFolder.cpp#265
This is in GetBackupSummaryFile method.
GetBackupSummaryFile is called here.
> 
> http://mxr.mozilla.org/comm-central/source/mail/base/content/folderPane.js#2012
> 2004       // Send a notification that we are triggering a database rebuild.
> 2012         folder.closeAndBackupFolderDB("");

closeAndBackupFolderDB is called at;
> http://mxr.mozilla.org/comm-central/search?string=CloseAndBackupFolderDB&find=&findi=&filter=^[^\0]*%24&hitlimit=&tree=comm-central
> Repair Folder
> http://mxr.mozilla.org/comm-central/source/mail/base/content/folderPane.js#2012
> Internal rebuild-index of local mail folder
> http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsLocalMailFolder.cpp#351
> renamr of IMAP folder
> http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapMailFolder.cpp#1673
> Folder re-initialzation due to "uid validity" change at server
> http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapMailFolder.cpp#2692

Following is MSGDB:5,MsgCopyService:5 log for "Repair Folder" of Inbox->No-Deliverd-To folder.
> 00000000	0.00000000	[4400] 0[230f140]: nsMsgDatabase::Open(C:\<Tb-prof>\Mail\Local Folders\Inbox.sbd\No-Deliverd-To.msf, FALSE, 8571bc0, TRUE)	
> 00000011	0.02753087	[4400] 0[230f140]: nsMsgDatabase::Open(C:\DOCUME~1\wada\LOCALS~1\Temp\MozillaMailnews\No-Deliverd-To.msf, FALSE, bc5dfa0, TRUE)	
> 00000022	0.05724247	[4400] 0[230f140]: closing database    C:\<Tb-prof>\Mail\Local Folders\Inbox.sbd\No-Deliverd-To.msf	
> 00000023	0.05900359	[4400] 0[230f140]: nsMsgDatabase::Open(C:\<Tb-prof>\Mail\Local Folders\Inbox.sbd\No-Deliverd-To.msf, TRUE, 8571bc0, TRUE)	
> 00000034	0.09233660	[4400] 0[230f140]: closing database    C:\DOCUME~1\wada\LOCALS~1\Temp\MozillaMailnews\No-Deliverd-To.msf

Following in http://www.edcint.co.nz/LargeFiles/Completed/5.Sentmultipletimes.9.TB.log perhaps corresponds to "Rebuild-Index of Sent folder of edcint.co.nz".
> 2013-08-27 21:55:27.177000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Sent.msf, FALSE, 4ec5f50, TRUE)
> 2013-08-27 21:55:27.177000 UTC - 0[e0f140]: error opening db 80550006
> 2013-08-27 21:55:27.193000 UTC - 0[e0f140]: closing database    S:\data\Mozilla Thunderbird\Matthew\edcint.co.nz\Sent.msf
> 2013-08-27 21:55:27.193000 UTC - 0[e0f140]: closing database    C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Sent.msf

Because "error open" is not logged for any actual message database, it may be following.
(1) Somehow Sent.msf of an account is broken, but,
  Sent.msf exists, file open of Sent.msf is successful,
  out-datedd condition doesn't exists,
so "error open db" is not logged.
Note: If file size=0 of Mbox.msf is forced(delete all data in Mbox.msf by Text Editor, "error opening db 80550005" is logged(NS_MSG_ERROR_FOLDER_SUMMARY_OUT_OF_DATE), so your case is not "file size=0 of Sent.msf".
(2) Because Sent.msf is broken, Rebuild-Index of Sent folder is invoked.
(3) When folder.closeAndBackupFolderDB is invoked by Rebuild-Index and backup database is created, ...\Temp\...\Trssh.msf is somehow not created, then "error open db" with NS_MSG_ERROR_FOLDER_SUMMARY_MISSING is logged upon the backup database open.

If "file open of file named Sent"(mail folder file, or offline-store file) for actual folder named Sent is interfered by other software including Tb's other process, icluding other software on SAMBA sever, read mail data fails or read mail data is not executed.
In this case, Rebuild-Index may be invoked without "error open db" log.
And "file not found on C:...\Temp\MozillaMailnews\Sent.msf" may be explained.

Anyway, if "Building summary ..." is observed often on Sent folder, get Process Monitor log for all Sent and Sent.msf under S: drive and C:...\Temp\MozillaMailnews, please.

FYI.
Concern on Trash and Sent.
- "Empty Trash" of local Trash folder is "delete Trash folder, then
   create new Trash folder".
- "Copy to Sent mail after send" is different from "append to Inbox
  by mail download", "filter move", "ordinal Copy/Move of mails".
  And, IIRC, "Copy to local Sent after mail send" deosn't open Sent
  folder explicitly. So, outdated-msf condition may occur frequently
  than other mail folder.

Comment 30

[WADA:World Anti-bad-Duping Agency]

Additional quick check result.
(1) Open Test.msf by Text Editor("open write" is used. not notepad.exe), and explicit folder open request by folder click.
"error opening db" is logged for both actual database and backup database.
  Actual Test.msf : 80550005 NS_MSG_ERROR_FOLDER_SUMMARY_OUT_OF_DATE
  Backup database : 80550006 NS_MSG_ERROR_FOLDER_SUMMARY_MISSING
C:...\Temp\MozillaMailnews\Test.msf was not deleted by Tb.
> 00000001	0.00068528	[5252] 0[230f140]: nsMsgDatabase::Open(C:\<tb-prof>\Mail\Local Folders\Test.msf, FALSE, 233c300, TRUE)	
> 00000002	0.00187007	[5252] 0[230f140]: error opening db 80004005	
> 00000009	0.00296909	[5252] 0[230f140]: closing database    C:\<tb-prof>\Mail\Local Folders\Test.msf	
> 00000010	0.00624521	[5252] 0[230f140]: nsMsgDatabase::Open(C:\<tb-prof>\Mail\Local Folders\Test.msf, FALSE, 233c300, TRUE)	
> 00000011	0.00744396	[5252] 0[230f140]: error opening db 80004005	
> 00000018	0.00882598	[5252] 0[230f140]: nsMsgDatabase::Open(C:\DOCUME~1\wada\LOCALS~1\Temp\MozillaMailnews\Test.msf, FALSE, 233c420, TRUE)	
> 00000019	0.00935510	[5252] 0[230f140]: error opening db 80550006	
> 00000026	0.00998507	[5252] 0[230f140]: closing database    C:\<tb-prof>\Mail\Local Folders\Test.msf	
> 00000027	0.01023203	[5252] 0[230f140]: closing database    C:\DOCUME~1\wada\LOCALS~1\Temp\MozillaMailnews\Test.msf	
> 00000028	0.02461235	[5252] 0[230f140]: nsMsgDatabase::Open(C:\DOCUME~1\wada\LOCALS~1\Temp\MozillaMailnews\Test.msf, FALSE, 233c300, TRUE)	
> 00000035	0.02899335	[5252] 0[230f140]: nsMsgDatabase::Open(C:\<tb-prof>\Mail\Local Folders\Test.msf, TRUE, 233c420, TRUE)	
> 00000036	0.03086985	[5252] 0[230f140]: error opening db 80520006	
> 00000043	0.03140539	[5252] 0[230f140]: closing database    C:\<tb-prof>\Mail\Local Folders\Test.msf	
> 00000044	0.38233045	[5252] 0[230f140]: nsMsgDatabase::Open(C:\<tb-prof>\Mail\Local Folders\Test.msf, FALSE, 233c540, TRUE)	
> 00000045	0.38360268	[5252] 0[230f140]: error opening db 80004005	
> 00000052	0.38480562	[5252] 0[230f140]: closing database    C:\<tb-prof>\Mail\Local Folders\Test.msf	
(2) Open Test by Text Editor("open write" is used. not notepad.exe), and explicit folder open by folder click.
(2-1) Mail view is normally executed. "Open read" by Tb looks successfull.
(2-2) Copy/Move some mails to Test.
Nothing happens, eEven though followig is logged by MsgCopyService:5.
> [2664] 0[230f140]: CopyMessages
> [2664] 0[230f140]: request 474aa40 CopyMessages request -
>   src mailbox://nobody@Local%20Folders/Unsent%20Messages
>   dest mailbox://nobody@Local%20Folders/Test numItems 0 type=0
> [2664] 0[230f140]: request 474aa40 DoCopy -
>   src mailbox://nobody@Local%20Folders/Unsent%20Messages
>   dest mailbox://nobody@Local%20Folders/Test numItems 1 type=0
No error is reported to Error Console, No log in Activity Manager.
No "error opening Db" because Test.msf is accessible.
"Repair Folder" is normally executed, and C:...\Temp\MozillaMailnews\Test.msf was normally deleted by Tb.

Because Rebuild-Index is not invoked by Filter Move, by Mail Copy/Move, when outdated-msf condition exists, "Rebuild Index upon Virtual Folder open" in your case may be next.
(1) Outdated-msf condition happens.
(2) Outdated-msf condition is detected, but Re-build-Index is not invoked automatically. And "Repair Folder is needed flag=On" is set.
(3) When folder is explicitly opened by Virtual Folder open, Rebuild-Index is invoked by  "Repair Folder is needed flag=On".
=> "error opening db" is not logged.

Comment 31

[WADA:World Anti-bad-Duping Agency]

Quick Check II(Tb 17 n Win-XP).
(1) Force outdated condition by "Increase file size of Test".
(1-1) Virtual Folder open, Search target folder = Test.
Following exception is reported to Error Console. However,"error opening db" is not logged nd Rebuild-Index is not invoked.
This occurs upon each Virtual Folder open.
> Error: Component returned failure code: 0x80550005 [nsIMsgDBView.open]
> Source File: resource:///modules/dbViewWrapper.js Line: 1055
> Error: Component returned failure code: 0x80550005 [nsIMsgFolder.msgDatabase]
> Source File: chrome://messenger/content/mailWidgets.xml Line: 2548
(1-2) Copy mails to Test.
Following Exception t Error Console, but no "error openng db" is logged.
> Error: 2013-08-29 14:29:14	moveCopyModule	ERROR	Exception:
>  [Exception... "'Component is not available' when calling method:
>  [nsIActivityManager::removeActivity]"
>  nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)"
>  location: "JS frame :: resource:///modules/activity/moveCopy.js ::
>  <TOP_LEVEL> :: line 133"  data: no]
> Source File: resource:///modules/gloda/log4moz.js Line: 687
Following is loged by MsgCopyservice:5. Mail data is normally appended to file named Test. But Rebuild-Index is ot ivoked auto-matically.
> [2664] 0[230f140]: NotifyCompletion -
>   src mailbox://nobody@Local%20Folders/Unsent%20Messages
>   dest mailbox://nobody@Local%20Folders/Test
> [2664] 0[230f140]: request 4747280 Clearing OK request -
>   src mailbox://nobody@Local%20Folders/Unsent%20Messages
>   dest mailbox://nobody@Local%20Folders/Test numItems 1 type=0
(1-3) Terminate Tb, restart Tb, and open Test(click at folder pane).
Following log sequence was observed.
  No "error opening db"error opening" for actual Temp.msf.
  "error opening db 80550006" for temprary database.
  (same as your log)  
> 00000000	0.00000000	[5420] 0[230f140]: nsMsgDatabase::Open(C:\<tb-prof>\Mail\Local Folders\Test.msf, FALSE, 233c420, TRUE)	
> 00000009	0.00339931	[5420] 0[230f140]: closing database    C:\<tb-prof>\Mail\Local Folders\Test.msf	
> 00000011	13.86120510	[5420] 0[230f140]: nsMsgDatabase::Open(C:\<tb-prof>\Mail\Local Folders\Test.msf, FALSE, 233c420, TRUE)	
> 00000019	13.86404610	[5420] 0[230f140]: closing database    C:\<tb-prof>\Mail\Local Folders\Test.msf	
> 00000020	13.86757469	[5420] 0[230f140]: nsMsgDatabase::Open(C:\<tb-prof>\Mail\Local Folders\Test.msf, FALSE, 233c420, TRUE)	
> 00000028	13.87048244	[5420] 0[230f140]: nsMsgDatabase::Open(C:\DOCUME~1\wada\LOCALS~1\Temp\MozillaMailnews\Test.msf, FALSE, 233c540, TRUE)	
> 00000029	13.87130260	[5420] 0[230f140]: error opening db 80550006	
> 00000037	13.87249565	[5420] 0[230f140]: closing database    C:\<tb-prof>\Mail\Local Folders\Test.msf	
> 00000038	13.87276649	[5420] 0[230f140]: closing database    C:\DOCUME~1\wada\LOCALS~1\Temp\MozillaMailnews\Test.msf	
> 00000039	13.88648224	[5420] 0[230f140]: nsMsgDatabase::Open(C:\DOCUME~1\wada\LOCALS~1\Temp\MozillaMailnews\Test.msf, FALSE, 233c420, TRUE)	
> 00000047	13.89161396	[5420] 0[230f140]: nsMsgDatabase::Open(C:\<tb-prof>\Mail\Local Folders\Test.msf, TRUE, 233c540, TRUE)	
> 00000055	13.94981766	[5420] 0[230f140]: closing database    C:\DOCUME~1\wada\LOCALS~1\Temp\MozillaMailnews\Test.msf	

It looks:
  1. Outdated-msf condition happens on Test.msf.
  2. When outdated-msf condition exists, filter move, mail etc. occurs.
     Rebuild-Index is not invoked.
  3. Terminate Tb => 
     Test.msf is initialized(identical to "no mail").
     Rimestamp/File size is correctly written to Test.msf
     => Outdated-msf condition is cleared.
        "Repair Folder is needed" tatus  may be written in Test.msf.
  4. Restart Tb, and open Test folder.
     Test.msf exists and is opened/read and no outdated-msf condition,
     so "error opening db" doesn't occur.
     "No mail" status in Test.msf, but "file size of Test">0.
     => Rebuild-index is invoked.
     Because of "No mail" status in Test.msf, 
     C:...\Temp\MozillaMailnews\Test.msf is not created.
     => "error opening db 80550006" for temprary database

By the way, flag for "Repair Folder is needed" was m_parsingFolder, and it's use was "if m_parsingFolder==true, return NS_MSG_ERROR_FOLDER_SUMMARY_OUT_OF_DATE to db access request, without additional file access".

Comment 32

[WADA:World Anti-bad-Duping Agency]

As seen in comment #31, when outdated-msf condition exists, if filter move, mail copy/move, virtual folder open, mouseover on folder etc. happens, exception may occur, and is logged to Error Concole if exception happens.
To bug opener: No such exception in Error Console while you are using Tb?

By the way, I found new method to force rebuild-index when .msf file is deleted.
  Apply columns to -> Folder and its children
Thread column setting is currently held in .msf file. Tb looks to internally open MsgDB upon saving Thread column setting.

Comment 33

[Matthew Jurgens]

No impact if C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews is deleted.
I am having trouble getting reproducible failures and collecting the information that you want. Perhaps this is a timing problem and when debug is off (normal mode) the timing hole exists and when debug is on and my computer is slow the timing hole is significantly smaller. Just a thought. I will keep trying to collect more information

Comment 34

[Matthew Jurgens]

Here is a procmon file that contains a rebuild of the Trash and Sent items folders.
http://www.edcint.co.nz/LargeFiles/Completed/8.Trash_Sent.PML

Comment 35

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #34)
> Here is a procmon file that contains a rebuild of the Trash and Sent items folders.

Trash and Spam, isn't it?

Following is seen.
(A) Spam
(A-1) Backup databace in %Temp% for "Spam.msf"
> 00:02:17.2318678	6:46:48.6166937	thunderbird.exe	C:\Documents and Settings\mjurgens\Local Settings\Temp\MozillaMailnews\Spam.msf	Read Metadata	IRP_MJ_DIRECTORY_CONTROL	NO SUCH FILE	Type: QueryDirectory, Filter: Spam.msf	5928	5932
> 00:02:17.2356058	6:46:48.6204317	thunderbird.exe	C:\Documents and Settings\mjurgens\Local Settings\Temp\MozillaMailnews\Spam.msf		FASTIO_NETWORK_QUERY_OPEN	NAME NOT FOUND		5928	5932
(A-2) Accessd file just before access to Temp\MozillaMailnews\Spam.msf
      S:\　... \Matthew\Local Folders\Spam.msf
      S:\　... \Matthew\Local Folders\Spam
> 00:01:05.8353921	6:45:37.2202180	thunderbird.exe	S:\Data\Mozilla Thunderbird\Matthew\Local Folders\Spam.msf	Read Metadata	IRP_MJ_QUERY_INFORMATION	SUCCESS	Type: QueryNetworkOpenInformationFile, CreationTime: 2013/09/01 21:12:08, LastAccessTime: 2013/09/01 22:12:32, LastWriteTime: 2013/09/01 21:12:08, ChangeTime: 2013/09/01 21:12:08, AllocationSize: 1601/01/01 9:00:00, EndOfFile: 1601/01/01 9:00:00, FileAttributes: A	5928	5932
> 00:01:05.8494164	6:45:37.2342423	thunderbird.exe	S:\Data\Mozilla Thunderbird\Matthew\Local Folders\Spam	Read Metadata	IRP_MJ_QUERY_INFORMATION	SUCCESS	Type: QueryBasicInformationFile, CreationTime: 2013/09/01 20:44:24, LastAccessTime: 2013/09/01 22:12:31, LastWriteTime: 2013/09/01 20:44:24, ChangeTime: 2013/09/01 20:44:24, FileAttributes: A	5928	5932

(B) Trash
(B-1) Backup databace in %Temp%
> 00:02:32.4839187	6:47:03.8687446	thunderbird.exe	C:\Documents and Settings\mjurgens\Local Settings\Temp\MozillaMailnews\Trash.msf	Read Metadata	IRP_MJ_DIRECTORY_CONTROL	NO SUCH FILE	Type: QueryDirectory, Filter: Trash.msf	5928	5932
> 00:02:32.5089393	6:47:03.8937652	thunderbird.exe	C:\Documents and Settings\mjurgens\Local Settings\Temp\MozillaMailnews\Trash.msf		FASTIO_NETWORK_QUERY_OPEN	NAME NOT FOUND		5928	5932
(B-2) Accessd file just before access to Temp\MozillaMailnews\Trash.msf
      S:\　... \Matthew\edcint.co.nz\Trash.msf
      S:\　... \Matthew\edcint.co.nz\Trash
> 00:01:07.6845801	6:45:39.0694060	thunderbird.exe	S:\Data\Mozilla Thunderbird\Matthew\edcint.co.nz\Trash.msf	Read Metadata	IRP_MJ_QUERY_INFORMATION	SUCCESS	Type: QueryNetworkOpenInformationFile, CreationTime: 2013/09/01 20:53:51, LastAccessTime: 2013/09/01 22:12:36, LastWriteTime: 2013/09/01 20:53:51, ChangeTime: 2013/09/01 20:53:51, AllocationSize: 1601/01/01 9:00:00, EndOfFile: 1601/01/01 9:00:00, FileAttributes: A	5928	5932
> 00:01:07.6994028	6:45:39.0842287	thunderbird.exe	S:\Data\Mozilla Thunderbird\Matthew\edcint.co.nz\Trash	Read Metadata	IRP_MJ_QUERY_INFORMATION	SUCCESS	Type: QueryBasicInformationFile, CreationTime: 2013/09/01 20:35:16, LastAccessTime: 2013/09/01 22:12:35, LastWriteTime: 2013/09/01 20:35:16, ChangeTime: 2013/09/01 20:35:16, FileAttributes: A	5928	5932

For these files, following is logged.
(1) xxx.msf is opened and data is read
    open/read pattern looks different between Spam.msf ad Trash.msf.
    - Spam.msf  : opened, full file is read,
                  opend again, part of data is re-read
    - Trash.msf : opened once, full file is read once
(2) Backup database
(2-a) Spam.msf is created, data is wriiten, dara is reead
(2-b) Trash.msf is not created
(3) file of xxx (not xxx.msf) is opened and full data is read
(4) content of xxx.msf is updated.
    It looks writing data at bottom of xxx.msf file.
    Perhaps MsgHdr information of newly detected mails
    by this Rebuild-Index.

There is is no "file open error". Mbox.msf and Mbox is opened normally and data is read normally, and Mbox.msf is updated normally.

If above is log for "rebuild-index by outdated-msf condition", log which you took was merely "file access log of rebuild-index when outdated-m,sf condition was already generated by someone".
Checking rraces/logs around "last update time of Spam or Trash".

(A) Spam
(A-1) Spam.msf
> 00:01:05.8353921	6:45:37.2202180	thunderbird.exe
>   S:\Data\Mozilla Thunderbird\Matthew\Local Folders\Spam.msf
>    Read Metadata	IRP_MJ_QUERY_INFORMATION	SUCCESS
>   Type: QueryNetworkOpenInformationFile,
>   CreationTime:   2013/09/01 21:12:08,
>   LastAccessTime: 2013/09/01 22:12:32,
>   LastWriteTime:  2013/09/01 21:12:08,
>   ChangeTime:     2013/09/01 21:12:08,
>   AllocationSize: 1601/01/01 9:00:00,
>   EndOfFile:      1601/01/01 9:00:00,
>   FileAttributes: A	5928	5932
(A-2) Spam
> 00:01:05.8494164	6:45:37.2342423	thunderbird.exe
>   S:\Data\Mozilla Thunderbird\Matthew\Local Folders\Spam
>   Read Metadata IRP_MJ_QUERY_INFORMATION  SUCCESS
>   Type: QueryBasicInformationFile,
>   CreationTime:   2013/09/01 20:44:24,
>   LastAccessTime: 2013/09/01 22:12:31,
>   LastWriteTime:  2013/09/01 20:44:24,
>   ChangeTime:     2013/09/01 20:44:24,
>   FileAttributes: A	5928	5932

(B) Trash
(B-1) Trash.msf
> 00:01:07.6845801	6:45:39.0694060	thunderbird.exe
>    S:\Data\Mozilla Thunderbird\Matthew\edcint.co.nz\Trash.msf
>    Read Metadata	IRP_MJ_QUERY_INFORMATION	SUCCESS
>    Type: QueryNetworkOpenInformationFile,
>    CreationTime:   2013/09/01 20:53:51,
>    LastAccessTime: 2013/09/01 22:12:36,
>    LastWriteTime:  2013/09/01 20:53:51,
>    ChangeTime:     2013/09/01 20:53:51,
>    AllocationSize: 1601/01/01 9:00:00,
>    EndOfFile: 1601/01/01 9:00:00,
>    FileAttributes: A	5928	5932
(B-2) Trash
> 00:01:07.6994028	6:45:39.0842287	thunderbird.exe
>   S:\Data\Mozilla Thunderbird\Matthew\edcint.co.nz\Trash
>   Read Metadata	IRP_MJ_QUERY_INFORMATION	SUCCESS
>   Type: QueryBasicInformationFile,
>   CreationTime:   2013/09/01 20:35:16,
>   LastAccessTime: 2013/09/01 22:12:35,
>   LastWriteTime:  2013/09/01 20:35:16,
>   ChangeTime:     2013/09/01 20:35:16,
>   FileAttributes: A	5928	5932

CreationTime==LastWriteTime in both Spam ad Trash. This is perhaps "file named Spam or Temp is re-created by automatic Compact".
What kind of data is logged by MSGDB:5 around LastWriteTime of Spam/Trash and CreationTime/LastAccessTime/LastWriteTime/ChangeTime of Spam.msf/Trash.msf?

If caused by Compact, problem may be following.
  Compact copies mail data from Mbox to nstmp and create nstmp.msf
  under directory where Mbox.msf is held.
  Timestamp/filesize of nstmp is wrtten in nstmp.msf.
  After it, Tb requests "move nstmp to Mbox with replace mode" and
  "move nstmp.msf to Mbox.msf with replace mode".
  When file named Mbox is opened just after the "move" operation
  (conceptually, identical to delete Mbox, rename nstmp to Mbox),
  file meta data of old Mbox file is returned from SMB1. 

By the way, what is reason to keep large Trash file size and Trash.msf file size?

 ,

Comment 36

[WADA:World Anti-bad-Duping Agency]

FYI.
To see process monitor log for specific mail folder files, define following as Filter rule of Process Monitor, and open saved PML file.
If you need to see log for Spam, check rules of Path for Spam and uncheck rules for Trash.
To quickly see File I/O by Tb, check "begin with  FASTIO_" only.
  Sequential entire file read with 32KB buffer(read of Mbox for rebuild-index), 
  Sequential entire file read with  8KB buffer(read of entire Mbox.msf after open),
  Non sequetial read, non sequential write,
  etc. can be observed easily.
  (For Mbox file, update of X-Mozilla-Status: is done by read and re-write)
When detailed information such as file meta data, check "begin with  IRP_MJ_" too.

   [ x ] Operation    begin with    FASTIO_    , Incluse
   [   ] Operation    begin with    IRP_MJ_    , Incluse

   [   ] Path         ends with     \Local Folders\spam      , Include
   [   ] Path         ends with     \Local Folders\spam.msf  , Include
   [   ] Path         contains      \MozillaMailnews\spam    , Include

   [ x ] Path         ends with     \edcint.co.nz\Trash      , Include
   [ x ] Path         ends with     \edcint.co.nz\Trash.msf  , Include
   [ x ] Path         contains      \MozillaMailnews\Trash   , Include

Comment 37

[Matthew Jurgens]

The 9.* files in http://www.edcint.co.nz/LargeFiles/Completed are for a TB run that included a Trash rebuild. In this case, trash was actually in such a state when TB started that filters could not actually move messages to the Trash. A dialog box popped up for every message to the effect of "I can't write to the Trash". After rebuilding the trash it worked ok.

Please remember a couple of things:
- This was never a problem in TB2. This problem started as soon as TB3 was released.
- This is only a problem once the profile/data files live on an SMB share. If they are on local disk it is never a problem.

My Trash is larger than you might be used to since I never empty it. When I have read a mail I just delete it and they stay there for reference. I do archive the trash. It normally only has a few months of emails in it.

Comment 38

[WADA:World Anti-bad-Duping Agency]

IIRC, a big change in Tb3 was;
  Till Tb 2, .msf file is not closed until shutdown.
  From Tb 3, .msf file is closed after a while.
  In Tb 1x.y, delay of DB close(physical .msf file write and file close) is changed.
As seen in Process Montor log pasted to comment #35, 
CreationTime===LastWriteTime===ChangeTime = 2013/09/01 20:mm:ss for all 4 files.
This indicates that all 4 files are created at 2013/09/01 20:mm:ss and is never modified by Tb since 2013/09/01 20:mm:ss. 
However, LastAccessTime is strange if it's by Tb.
  File meta data at 2013/09/02 6:45:37 or 6:45:39.
                                                              LastAccessTime:
  S:\Data\Mozilla Thunderbird\Matthew\Local Folders\Spam      2013/09/01 22:12:31
  S:\Data\Mozilla Thunderbird\Matthew\Local Folders\Spam.msf  2013/09/01 22:12:32
  S:\Data\Mozilla Thunderbird\Matthew\edcint.co.nz\Trash      2013/09/01 22:12:35
  S:\Data\Mozilla Thunderbird\Matthew\edcint.co.nz\Trash.msf  2013/09/01 22:12:36
This access is perhaps by file backup tool on File Server.

If problem was "interfere of Tb's .msf file open by backup tool when Tb tried to do filter move etc.", and if backup tool opens files sequentially instead of concurrently, Modified timestamp of Spam or Trash should be updated, because Tb opens Spam or Trash even when open of Spam.msf or Trash.msf fails, and Tb appends mail data to Spam or Trash.
So, I guess that update of Spam folder and Trash folder by Tb didn't occur since 2013/09/01 20:mm:ss.

If it's true, it may be by file timestamp chaos in MS Win.
  IIRC, MS Win returns Last Access Time as Last Modified Time
  to DIR command or something, for compatibility of MS DOS application.
If LastAccessTime is passed from OS as "last modified time" to Tb, "delta of file timestamp > timestamp_leeway(default=4000, perhaps 4000sec)" can occur.
Can you check following?
(1) Create a local mail folder on S: drive(call Test), and copy some mails to Test.
(2) Terminate Tb to force close of Test.msf/Test.
(3) Retart Tb, and don't touch Test folder.
(4) After 2 hours, change LastAccessTime of Test(and Test.msf if required) by "read open/close of Test" or by touch command at File Server.
(5) Open Test folder by Tb.
Is Rebuild-Index invoked?

Comment 39

[WADA:World Anti-bad-Duping Agency]

FYI.
Type: of IRP_MJ_QUERY_INFORMATION is different between .msf and non-msf file in Process Monitor log.
  .msf file    : QueryNetworkOpenInformationFile
  non msf file : QueryBasicInformationFile
> S:\Data\Mozilla Thunderbird\Matthew\edcint.co.nz\Trash.msf
>    Read Metadata	IRP_MJ_QUERY_INFORMATION	SUCCESS
>    Type: QueryNetworkOpenInformationFile,
> S:\Data\Mozilla Thunderbird\Matthew\edcint.co.nz\Trash
>    Read Metadata	IRP_MJ_QUERY_INFORMATION	SUCCESS
>    Type: QueryBasicInformationFile,
This may be merely difference between Read/Write open and Read open, but this may be difference between API used by Tb for file access. If latter, it may be relivant to problem because problem is relivant to file meta data of non-msf file placed at SMB1 server.

Comment 40

[WADA:World Anti-bad-Duping Agency]

Another concern : Auto-Cmpact.

> Trash :
>  S:\Data\Mozilla Thunderbird\Matthew\edcint.co.nz\Trash
>    CreationTime: == LastWriteTime: ==  ChangeTime: = 2013/09/01 20:35:16
>  S:\Data\ ... \Matthew\edcint.co.nz\Trash.msf
>    CreationTime: == LastWriteTime: ==  ChangeTime: = 2013/09/01 20:53:51
> Spam :
>  S:\Data\ ... \Matthew\Local Folders\Spam
>    CreationTime: == LastWriteTime: ==  ChangeTime: = 2013/09/01 20:44:24
>  S:\Data\ ... \Matthew\Local Folders\Spam.msf
>    CreationTime: == LastWriteTime: ==  ChangeTime: = 2013/09/01 21:12:08
If CreationTime: is correct, and if Spam/Trash file was created by Tb istead of by "backup/restore", Spam/Trash file was generated by Compact of Tb.
If so, timestamps indicates;
> 2013/09/01 20:xx:yy : Auto-Compact is invoked without confrmation dialog before auto-compact. 
> 2013/09/01 20:35:16 : non-deleted mails are copied from Trash to nstmpX, and nstmpX is closed,
>.                      and "move nstmpX Trash(with replace)" is requested, then nstmpX is renamed to Trash.
> 2013/09/01 20:44:24 : Same thing as one on Trash occurs on Spam, and nstmpY is created and moved to Spam.
> 2013/09/01 20:53:51 : nstmpX.msf is created from mail data in nstmpX, and nstmpX.msf is closed,
>                       and "move nstmpX.msf Trash.msf(with replace)" is requested, then nstmpX.msf was renamed to Trash,
>                       Or, MsgDatabase is re-created in memory from mail data in nstmpX,
>                       and MsgDatabase was closed after 30 minutes,
>                       and Trash.msf was physically written to Trash.msf.
> 2013/09/01 21:12:08 : Same thing as Trash.msf occurs on Spam.msf
>                       after 30 minutes from Spam file creation.
After above, it looks that Tb never modified Trash file, Tras.msf file, Spam file, Spam.msf file, until next morning.
So, "outdated-msf condition" may be generated upon avove Auto-Compact of Trash and Spam.
If file meta data of Trash/Spam is requestd by Tb just after "move nstmpX Trash" / "move nstmpY Spam", old file meta data before "move with replace" from SMB1 client on PC.

If possible, get Process Monitor log for "auto-compact of a small mail folder", please
(1) mail.prompt_purge_threshhold = true
    mail.purge_threshhold_mb     = mm
    mail.purge.ask = true (after change to true, restar of Tb is mandatory)
(2) Create a dummy POP3 account(server=z.z.z,userID=any ascii string,mail-addr is dummy),
    with mail directory on S: drive, with any new mail checking disabled.
    Restart of Tb is needed after "Local Directory:" change.
    Create a local folder under z.z.z(call ZZZ), copy some mails to ZZZ,
    and delete(Shft+Delete) several mails from ZZZ in order to force Compact by auto-compact.
(3) Restart Tb, don't touch ZZZ folder.
(4) Start Process Monitor
    Filter example :
       Path doesn't contain \Matthew\z.z.z\  , Exclude
       Path contains        ZZZ              , Include
       Path contains        nstmp            , include
(5) Use Tb as usual.
    If possible, delete many mails or move many spams to invoke auto-compact quickly,
    or use smaller mail.purge_threshhold_mb for test.
(6) When dialog before auto-compact is shown, reply OK.
(7) After end of "Compact of ZZZ by auto-compact", terminate Process Monitor.

Note:
If cause is "auto-compact of folder on file server", and if "outdated-msf condition" was already implemented in Tb 2, I think problem should have occurred in Tb2 too.

Comment 41

[WADA:World Anti-bad-Duping Agency]

Confirming per MSGDB:5 log and Prcess Monitor log.

Comment 42

[WADA:World Anti-bad-Duping Agency]

In Linux, CreationTime: was not supported by file system.
> http://linux.die.net/man/2/stat
> http://unix.stackexchange.com/questions/24441/get-file-created-creation-time
>   Only LastAccessTime, LastWriteTime, ChangeTime, are used in file system of Linux.
>   (following is also seen in this page)
>   The file creation time is actually stored in Ext4,
>   but not directly accessible.
So, my assumptions based on CreationTime: value was wrong. Sorry for my confusions.

Following is document for "File Times" in MS's file system.
> http://msdn.microsoft.com/en-us/library/windows/desktop/ms724290%28v=vs.85%29.aspx
IIRC, smbfs had similar issue in FAT, but you say "CIFS share" in comment #16.
It may be "cached file meta data in SMB1" relevant issue on file size instead of file time.

Comment 43

[WADA:World Anti-bad-Duping Agency]

In 9.TB.log obtained from 2013-09-02 09:51:34, "nsMsgDatabase::Open(C:\DOCUME~1\..." is seen also for "S:\data\...\edcint.co.nz\Sent.msf" and "S:\data\...\edcint.co.nz\Bulk.sbd\Nagios-Users.msf", in addition to "S:\Data\...\Local Folders\Spam.msf" and "S:\Data\...\edcint.co.nz\Trash.msf".
Why problem repeatedly occurs on these four mail folders?
What is difference of these folders from other folders? Big file size only?
Files for these folder are not placed under profile directory.
> (i) profile on network drive(with mail directory under profile)
> > S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\smart mailboxes\Inbox.msf
> > S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\pop3.edcint.co.nz\Inbox.msf
> (ii) mail directory on network drive which is different from profile directory.
> > S:\data\Mozilla Thunderbird\Matthew\edcint.co.nz\Inbox.msf
> > S:\data\Mozilla Thunderbird\Matthew\Local Folders\Inbox.msf
Is it relevant?
What is reason to hold mail directory for dcint.co.nz and Local Folders at highger level than Tb's profile directory, even though you changed mail directory for "smart mailboxes" to S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\smart mailboxes? Shared between multiple Tb instances?

Anyway, to know what happened, al least MSGDB:5 log and Process Monitor log from "LastWriteTime/ChangeTime of Trash/Trash.msf or Spam/Spam.msf" to "Rebuild-Index time by outdated-msf condition" at Client PC.
To know "when outdated-msf condition was generated in .msf file", "traces/logs before Rebuild-Index happened due to outdated-msf condition" is needed.
And, file access log for Trash/Trash.msf or Spam/Spam.msf at file server is needed if file access by server application is relevant to problem.

Comment 44

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #37)
> - This is only a problem once the profile/data files live on an SMB share.
> If they are on local disk it is never a problem.

Biggest difference between local files and files live on an SMB share.
- Local files : no one on other PC can access the file, unless PC user opens file to
                public.
- SMB files : Any program on any other PC can access your directories/files,
              as far as SMB server is reachable and someone knows "file share name"
              of your shared directory.
This is reason why I repeatedly asked you about "interfer of other software on other PC".
If other software on other PC opens files under "S:\data\Mozilla Thunderbird\Matthew\", and if LAN cable is pulled at the PC, file lock is held for long time at SMB1 server. If this happens on "S:\data\Mozilla Thunderbird\Matthew\Local Folders\Trash.msf", "Read/Write open of ...\Trash.msf by Tb" is interfered for long time/multiple times. This can easily produce outdated-msf condition of the Trash folder.
Please surely rule out "interfere of other software on other PC".

Comment 45

[Matthew Jurgens]

(In reply to WADA from comment #38)
> Is Rebuild-Index invoked?
No.

Before this test I disabled all backups. No backups ran during this period. After 2 hours (TB was running the whole time between step 3 and step 5)the last mod time of the "test" files were unchanged. I "touched" the files and the last mod time was updated. No folder rebuild occurred. However, a Trash and Sent items folder rebuild did occur. Remember that during this period there was nothing touching the TB files except for TB and TB was running the whole time. The rebuild occured when I accessed my virtual folder called "Search to Rebuild Folder". This is virtual folder that drives a search that finds all mail less than 1 day old. I use this as an easy way to touch all folders - this automatically rebuilds folders that it needs to - way easier than manually trying to find folders that need rebuilding.

I have left my backups disabled so that nothing else will touch the TB files except for TB. I will leave this disabled for a day but I think I have already proved that there is nothing else touching the files and yet they still rebuild themselves.

Comment 46

[Matthew Jurgens]

(In reply to WADA from comment #43)
> Why problem repeatedly occurs on these four mail folders?
> What is difference of these folders from other folders? Big file size only?
These are probably that most frequently changed folders.

> Files for these folder are not placed under profile directory.
This is done on purpose. Default TB install has mail data separate to profile data and so do I.

> > (i) profile on network drive(with mail directory under profile)
> > > S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\smart mailboxes\Inbox.msf
> > > S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\pop3.edcint.co.nz\Inbox.msf
No idea. I did not create these folders. TB must be creating them.

> What is reason to hold mail directory for dcint.co.nz and Local Folders at
> highger level than Tb's profile directory, even though you changed mail
> directory for "smart mailboxes" to S:\data\Mozilla
> Thunderbird\Profiles\Matthew\Mail\smart mailboxes? Shared between multiple
> Tb instances?
I never changed "smart mailboxes" location.
Reason for installing on a shared drive is so I can easily back it up.

Comment 47

[Matthew Jurgens]

(In reply to WADA from comment #40)
> Another concern : Auto-Cmpact.
> 
I don't think this is a concern since the problem happens if Auto-compact is disabled or enabled.

Comment 48

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #46)
> > Files for these folder are not placed under profile directory.
> This is done on purpose. Default TB install has mail data separate to profile data and so do I.
> > > (i) profile on network drive(with mail directory under profile)
> > > > S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\smart mailboxes\Inbox.msf
> > > > S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\pop3.edcint.co.nz\Inbox.msf
> No idea. I did not create these folders. TB must be creating them.

Aha!
You defined edcint.co.nz(and Local Folders) before transfer to "profile on S: drive". You changed "Local Directory:" to "S:\data\Mozilla Thunderbird\Matthew". Or, when transfer to "profile on S: drive", you moved or copied "C:\...\Mail" to "S:\data\Mozilla Thunderbird\Matthew", and changed "Local Directory:" setting properly.
And, you created pop3.edcint.co.nz account etc. after transfer to "profile on S:", while Tb is running with profile directory="S:\data\Mozilla Thunderbird\Profiles\Matthew".

Big difference between "S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail" and "S:\data\Mozilla Thunderbird\Matthew" is;
  S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail : many mail folders of many accounts
  S:\data\Mozilla Thunderbird\Matthew               : pretty limited files for mail folder
=>
  S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail :
    Because many files are opened(accounN\Inbox.msf, ... for N=1 to M)
    and mails are downloaded to ...\accounN\Inbox at many accounts,
    connection for S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail is always active.
  S:\data\Mozilla Thunderbird\Matthew :
    Because repeated access is limited to ...\edcint.co.nz\Inbox.msf(and Inbox) only
    which is invoked by new mail download to Inbox at edcint.co.nz,
    connection for S:\data\Mozilla Thunderbird\Matthew may be dropped
    with keeping session for files.
    => When reconnection is needed for file open requet,
       something wrong may occur on file meta data.

Do you see problem frequently on edcint.co.nz\Spam.msf and Local Folders\Trash.msf with "mail directory under profile directory on S: drive"?
(1) Copy S:\data\Mozilla Thunderbird\Matthew\edcint.co.nz
      to S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz
    Copy S:\data\Mozilla Thunderbird\Matthew\Local Folders
      to S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders
(2) At Server Settings, "Local Directory:" field, change mail directory location,
    and restart Tb. ("Restart Tb after change" is mandatory)
      edcint.co.nz  : S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz
      Local Folders : S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders
    This is identical to serverX.dir-rel=[ProfD]Mail/edcint.co.nz(Local Folders).

Comment 49

[Matthew Jurgens]

> Aha!
> You defined edcint.co.nz(and Local Folders) before transfer to "profile on
> S: drive". 
It is now a mixture. Some accounts have been added after the move from the local directory.

> Do you see problem frequently on edcint.co.nz\Spam.msf and Local
> Folders\Trash.msf with "mail directory under profile directory on S: drive"?
Spam not so much, Trash rebuilds are probably the most common (probably the most commonly used folder) but I can not tell which Trash folder from which account is the problem most often.


> (1) Copy S:\data\Mozilla Thunderbird\Matthew\edcint.co.nz
>       to S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz
>     Copy S:\data\Mozilla Thunderbird\Matthew\Local Folders
>       to S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders
So you want me to change the data folder to be under the same path as the profiles?

Comment 50

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #49)
> So you want me to change the data folder to be under the same path as the profiles?

Yes.
Difference among following may help problem determination.
(a) both mail directories are placed outside of profile directory
(b) both mail directories are placed under profile directory
(c) one mail directory is placed outside of profile directory,
    another mail directory is placed under profile directory
(changing back to original is pretty easy. Simply do "copy back directory, change Local Directory: setting, restart Tb")
And, check with mail.db_timestamp_leeway=172800(3600*24*2, perhaps 48hours=2days) will be perhaps helpful to rule out "timestamp_leeway only problem".

Get MSGDB:5 log and Process Monitor log from "Intentional Repair Folder"(surely no outdated-msf condition) till when you are aware of "outdated-msf condition" by Virtual Folder open, Delete mails, folder open by click folder(rebuild-index is invoked if outdated-msf condition exists), exception in Error Console upon mouseover to folder, "Building summary..." message and so on", please.

Because it looks that problem frequently occurs on Spam, Trash, Sent in your environment, use following Filter in Process Monitor, in order to reduce file size of saved log.
  Check following if outside of profile directory.
  [   ] Path contains \Matthew\Local Folders\Spam  , Include
  [   ] Path contains \Matthew\edcint.co.nz\Sent   , Include
  [   ] Path contains \Matthew\edcint.co.nz\Trash  , Include
  Check following if placed under profile directory.
  [ x ] Path contains \Profiles\Matthew\Mail\Local Folders\Spam  , Include
  [ x ] Path contains \Profiles\Matthew\Mail\edcint.co.nz\Sent   , Include
  [ x ] Path contains \Profiles\Matthew\Mail\edcint.co.nz\Trash  , Include

Comment 51

[Matthew Jurgens]

Attachment: cantwritetofolder.png

This is the dialog box I have started seeing just after TB startup as it starts to download POP mail and trys to filter it to various folders. This started happening with TB Beta 23.

Comment 52

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #51)
> cantwritetofolder.png

> This is the dialog box I have started seeing just after TB startup
> as it starts to download POP mail and trys to filter it to various folders.

This kind of message is shown when Mbox, Mbox.msf file open error, Mbox, Mbox.msf file write error. 

> This started happening with TB Beta 23.

This is perhaps *improvenent* in Tb 23.
  Befere Tb 23 : When outdated-msf condition is detected, filter move ignored it.
  After  Tb 23 : error message is shown when filter move detected outdated-msf condition.
If so, your problem occurred on \Matthew\Local Folders\Spam.msf as usual and filter move of Tb 23 detected already generated outdated-msf condition, or file I/O error on Spam.msf/Spam started to occur just before first filter move was attempted after restart of Tb 23. 

Outdated-msf condition was perhaps generated before termination of Tb and the restart of Tb using Tb 23. When was first "error opening db" logged by MSGDB:5, after you confirmed that outdated-msf condition surely doesn't exist, during previous run of Tb? What kind of log is seen in Process Monitor log around the first "error opening db" log?

Comment 53

[WADA:World Anti-bad-Duping Agency]

(In addition to comment #50)
Add following Filter rule for "backup database used by Rebuild-Index" upon Process Monitor log save, please.
>   [ x ] Path contains \MozillaMailnews\Spam   , Include
>   [ x ] Path contains \MozillaMailnews\Sent   , Include
>   [ x ] Path contains \MozillaMailnews\Trash  , Include

Comment 54

[Matthew Jurgens]

> > (1) Copy S:\data\Mozilla Thunderbird\Matthew\edcint.co.nz
> >       to S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz
> >     Copy S:\data\Mozilla Thunderbird\Matthew\Local Folders
> >       to S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders
I have changed the above folder locations. Is the intent to change all my mail data to under the profile directory? The reason I ask is that there is more data to move if that is the intent.

Comment 55

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #54)
> > > (1) Copy S:\data\Mozilla Thunderbird\Matthew\edcint.co.nz
> > >       to S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz
> > >     Copy S:\data\Mozilla Thunderbird\Matthew\Local Folders
> > >       to S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders
> I have changed the above folder locations.

Thanks.

> Is the intent to change all my mail data to under the profile directory?
> The reason I ask is that there is more data to move if that is the intent.

No.
Because your problem looks to occur frequently on Trash or Spam of these 2 accounts, I simply asked you to check difference between "mail folder file is placed under profile directory" and "mail folder file is placed outside of profile directory" for problem determination.

How can we say something on "when and how and why outdated-msf condition was generated" ｂｙ log/trace data for when "rebuild-index was finally invoked by　detection of already genarated outdated-msf condition"?

Comment 56

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #2)
> SMB1.

What is set in "server max protocol" of smb.conf of your SAMBA server?
> http://www.samba.org/samba/docs/man/manpages-3/smb.conf.5.html#SERVERMAXPROTOCOL
It's perhaps NT1, because you say SMB1.
Will "server max protocol = LANMAN1"(most primitive one) be a workaround of your problem?

Can you check with "oplock = no"?
> http://www.samba.org/samba/docs/man/manpages-3/smb.conf.5.html#OPLOCKS

What is set in following? (perhaps Default is used)
 strict locking, strict sync, sync always 

Because "create time for a file" is stored by "store dos attributes = yes", it may make problem analysis by "Process Monitor log" easier.

Comment 57

[WADA:World Anti-bad-Duping Agency]

Ineresting document was found by Google Search for "samba cifs file timestamp".
> http://www.nfsv4bat.org/Documents/nasconf/2005/allison.pdf
According to this document "timestamp of actual file in File System of Sever, on HDD of servrt(Linux in your case)" seems "timestamp when the actual file is closed by SMB file share program(corresponds to Lan Manager etc.)".

From perspective of OS and File System, this is pretty normal for me, because the actual file on HDD is shared by Lan Manager on Linux server and other applications on same Linux server, and Lan Manager opens the actual file for SMB file sharing, updates file content according to request from SMB client, and finally closes the actual file on HDD upon Lan Manager termiation. "File shared via SMB among SMB clients" is an "virtual file managed by SMB modules within SMB", and Server OS and File System of Server doesn't know about the "virtual file within SMB".

If this is true, timestamp_leeway=4000 may be too small in some environments.
  - Big timestamp difference may occur.
    - By summer time, local time may differ one hour.
    - Because of caching in NTFS(held in cache for max one hour),
      file timestamp of file in file system(on HDD) may differ one hour
      from timestamp which Tb obtained from SMB upon Mbox file close.
  - Small timestamp difference may occur.
    After Tb closed SMB shared Mbox file,
    physical file close of actual/physical Mbox file in file system, on HDD,
    is executed by SMB module after a while. This "a while" is perhaps not so small.

Matthew Jurgens, please set mail.db_timestamp_leeway=172800(3600*24*2, perhaps 48hours=2days) in your daily use of Tb, in order to rule out "outdated-msf condition due to timestamp_leeway only, no file size issue".
File size has higher priority, then outdated-msf condition occurs if file size mismatch exists, regardless of timestamp value. So, big mail.db_timestamp_leeway won't produce any problem, unless someone alters your Mbox file content without file size change.

Comment 58

[Matthew Jurgens]

> What is set in "server max protocol" of smb.conf of your SAMBA server?
SMB2

> Will "server max protocol = LANMAN1"(most primitive one) be a workaround of
> your problem?
I can try this later (after trying oplocks).

> Can you check with "oplock = no"?
> > http://www.samba.org/samba/docs/man/manpages-3/smb.conf.5.html#OPLOCKS
I have set this now

> What is set in following? (perhaps Default is used)
>  strict locking, strict sync, sync always 
Default values as they are not set specifically

> Because "create time for a file" is stored by "store dos attributes = yes",
> it may make problem analysis by "Process Monitor log" easier.
I have set this now

Comment 59

[Matthew Jurgens]

> Matthew Jurgens, please set mail.db_timestamp_leeway=172800(3600*24*2,
done

Comment 60

[script]

I ahve the same issue with my profile in a single folder on D:/ TB 24.0 Win7 db_timesampt_leeway=172800.

A specific subset of my folders' msf is rebuilt every time I

1) restart TB
2) CTRl-SHIFT-D search local folders

The subset of folders that has its MSF rebuilt is a few years old, from 2009 as verified by their MSWindows "Date modified", 70MB size. As expected the rebuilt msfs are md5sum identical.

The problem first occurred about half a year ago.

Comment 61

[WADA:World Anti-bad-Duping Agency]

(In reply to script from comment #60)
> I ahve the same issue with my profile in a single folder on D:/ TB 24.0 Win7
> db_timesampt_leeway=172800.

To script. What is SAMBA version in your case? What is your smb.conf settings? Have you tried smb.conf settings written in comment #58?

Comment 62

[script]

(In reply to WADA from comment #61)
> To script. What is SAMBA version in your case? What is your smb.conf
> settings? Have you tried smb.conf settings written in comment #58?

My issue occurs even WITHOUT using a network drive, on my MSWindows7 D:/
This bug was the closest I could find and maybe related.

Comment 63

[WADA:World Anti-bad-Duping Agency]

(In reply to script from comment #62)
> My issue occurs even WITHOUT using a network drive, (snip)

Sorry but this bug is for "outdated-msf condition with SMB1 file share" only. This bug is never for problem with profile on local hard disk, even though outdated-msf condition can occur in both "profile on network drive" and "profile on local drive".
Please search bugzilla.mozilla.org well for your problem.

FYI.
If profile on local drive(mail directory on local drive), it should work with db_timesampt_leeway=0 except when very rare condition, such as power failure in critical situation, and test with large db_timesampt_leeway value is usually meaningless.

Comment 64

[kev]

We're experiencing this with Samba 4.0.9 and TB 24.0 (also on 17.0.8, which wasn't auto-upgrading for some reason.)

smb.conf is:

read only = No
create mask = 0777
force create mode = 0777
directory mask = 0777
force directory mode = 0777

Comment 65

[WADA:World Anti-bad-Duping Agency]

In Tb 24, "error opening db 8055000?" for actual MsgDatabase(.msf file) was not observed in MSGDB:5 log. Only "error opening db 80550006" for Backup Database(...\Temp\MozillaMailnews\<MboxName>.msf) was seen by following test.
(1) Add null lines(CRLF) to file name Test(size is changed, timestamis changed but difference is less than timestamp_leeway=4000).
(2) Restart Tb, with MSGDB:5,MsgCopyService:5
(3) Repeat "copy or move or append unread messages to folder named Test" several times.
(3-1) Move unread messages by message filter
(3-2) Copy unread messages manually(via Menu, not by Drag&Drop)
(3-3) Save draft mail, with setting test as "draft folder"
In any case, "error opening db" is not logged, and log by MsgCopyService:5 is written, and message of "nn mails are copied" was shown at Activity manager.
Message data is normally appended file named Test.
File named Test.msf is not physically updated, even when "closing database ...\Test.msf is logged by MSGDB:5.
Unread mail cout at folder pane is not updated.
(4) Open Search folder(search target folder=Test only)
    "error opening db" is not logged.
    Thread pane rows are shown, but nothing is shown at thread pane.
      <= MsgDatabase(.msf data in memory) is perhaps cleared, with number of mails kept)
(5) Open Folder Property of Text, and Cancel
    "error opening db" is not logged for actual Test.msf.
    Open log for ...\Temp\MozillaMailnews\Test.msf is written.
    "error opening db 80550006" is written for ...\Temp\MozillaMailnews\Test.msf.
    => Internal rebuild-index is perhaps invoked.
    Unread message count of Test is not updated at folder pane.
(5) Open Search folder(search target folder=Test only) again.
    => Search result is normally shown.

As seen in above test result, "error opening db" doesn't look written for actual MsgDatabase. 
i.e.
Once outdated-msf condition happens, filter move/message copy/message move/draft save/copy to Sent appends mail data to Test normally(any is executed via MsgCopyService), but doesn't invoke rebuild-index8known bug) and does keep btoken status of ".msf data in memory". move.copy/save is reported as "successful" by MsgCopyService to caller.
Rebuild-Index is invoked only by Folder click or Folder Property open.

Because of this, "when outdated condition was generated" can not be known by MSGDB:5,MsgCopyService:5 logging. It's merely helpful to know "rebuild-index was actually executed". I believe debug build is needed.

I'm now suspecting "interfere of Tb's .msf file open by backup software at server" again.
Because "filter move of Tb" currently doesn't report error even when .msf file open fails, both Process Monitor log at client(Win-XP) and file access monitor log at sever are needed.

Comment 66

[WADA:World Anti-bad-Duping Agency]

FYI.
When filter move was requested while Test.msf was opened by other prgram(open write exclusive), following log was written by MSGDB:5.
> nsMsgDatabase::Open(C:\Documents and Settings\ ... \Mail\Local Folders\TestX.sbd\TestY.sbd\Test.msf, FALSE, 735cec0, TRUE)
> error opening db 80004005
Mail data is normally appened to file named Test, so outdated-msf condition may be generated.

If this kind of problem, NSPR log by MSGDB:5,MsgCopyService:5 is helpful in trouble shooting.

Comment 67

[kev]

(In reply to WADA from comment #65)
> I'm now suspecting "interfere of Tb's .msf file open by backup software at
> server" again.

If you mean this in general, I'd like to note that in our case, Tb clients start up in the morning and experience frequent rebuilds throughout the day, whereas our server-side backup only runs at night.  Not saying that's never it, but probably not the reason in our setup.

Comment 68

[Matthew Jurgens]

> I'm now suspecting "interfere of Tb's .msf file open by backup software at
> server" again.

I turned off all my backups for 2 days and rebuilds still happened during that period. The biggest step towards fixing this problem is running procmon before starting TB and then running TB with debugging turned on. TB runs so slowly with all the debugging and the problem hardly ever happens.

This makes me think it is some kind of timing hole.

I am currently running with a leeway of about 172000 although I noticed the less frequent rebuilds previously running all the debug with a leeway of 4000 as well.

Comment 69

[kev]

A user on our network set the leeway to 172800 and restarted Tb 24.0, but her sent box is already rebuilding.

Comment 70

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #68)
> > I'm now suspecting "interfere of Tb's .msf file open by backup software at
> > server" again.
> I turned off all my backups for 2 days and rebuilds still happened during
> that period. The biggest step towards fixing this problem is running procmon
> before starting TB and then running TB with debugging turned on. TB runs so
> slowly with all the debugging and the problem hardly ever happens.

Process Monitor keeps log data in Virtual Memory by default. It may produce expand of swap file, and it may produce slow down if Process Monitor is kept open for long time. File, Backing Up, is option for "where log is kept". By setting it to HDD file, you can see log file size, and you can know that it's pretty huge.
Please stop Process Monitor in your daily use. Please enable Tb's NSPR logging only(timestamp,MSGDB:5,MsgCopyService:5) in your daily use.

> This makes me think it is some kind of timing hole.

It may be interfere between Tb's tasks. For example,
  POP3 account#1 downloads new mails, and filter moves mails to \Local Folders\Mbox.
  At same time, POP3 account#2 downloads new mails, and filter moves mails to same Mbox.
If Mbox is on local HDD, each "move mails to Mbox" is not so slow. However, if network drive, each "move mails to Mbox" takes pretty long, because one write request is issued by Tb for each mail data line(known bug).

Do you have such message filter rules?

Do you see MSGDB:5 log like "error opening db 80004005"(reason code is not 80550004 nor 80550006 which are for outdated-msf condition)?
If such log is seen in log file, what log is written by MsgCopyService:5 around it?
If such log is not seen in log but rebuild-index happens, is log by MsgCopyService:5 written for same destination folder concecutively?

Comment 71

[kev]

Sometimes, but not in every case, a rebuild causes the files to be written with a different user:group and unix ACL.  E.g. today's ls -lt reports:

-rwxrwxrwx  1    1128     513      27808 Sep 27 16:00 Inbox.msf
-rwxrwxrwx  1    1128     513      21986 Sep 27 16:00 popstate.dat
-rwxrwx---+ 1 3000000 3000001   20668851 Sep 27 16:00 Sent.msf
-rwxrwxrwx  1    1128     513 2490510910 Sep 27 16:00 Sent
-rwxrwxrwx  1    1128     513    3305947 Sep 27 15:59 Inbox
-rwxrwxrwx  1    1128     513       7562 Sep 27 15:28 Drafts.msf
-rwxrwxrwx  1    1128     513  110695482 Sep 27 15:24 Drafts
-rwxrwxrwx  1    1128     513   99108571 Sep 27 15:08 Trash
-rwxrwxrwx  1    1128     513     243833 Sep 27 15:08 Junk
-rwxrwx---+ 1 3000000 3000001       4951 Sep 27 15:08 Junk.msf
-rwxrwxrwx  1    1128     513  461630176 Sep 27 15:04 Internal
-rwxrwx---+ 1 3000000 3000001    4391859 Sep 27 15:04 Internal.msf
-rwxrwx---+ 1 3000000 3000001    7595562 Sep 27 14:46 Vendor Info.msf
-rwxrwxrwx  1    1128     513    1658123 Sep 27 14:30 Customer Info.msf
drwxrwxrwx. 5    1128     513       4096 Sep 27 14:30 Internal.sbd
-rwxrwxrwx  1    1128     513       9099 Sep 27 14:27 Templates.msf

Not sure if this is related or not?

Comment 72

[kev]

Although, another user shows even the mailboxes (not just the .msf summaries) have this (also seemingly randomly, if not time-wise, at least as to which ones):

-rwxrwx---+ 1 3000000 3000001   10346701 Sep 27 12:41 Trash.msf
-rwxrwx---+ 1 3000000 3000001      14269 Sep 27 12:41 Inbox.msf
-rwxrwx---+ 1 3000000 3000001      10597 Sep 27 12:41 Drafts.msf
-rwxrwx---+ 1 3000000 3000001 1052436205 Sep 27 12:41 Trash
-rwxrwx---+ 1 3000000 3000001     154681 Sep 27 12:41 Inbox
-rwxrwx---+ 1 3000000 3000001    7146482 Sep 27 12:40 Sent.msf
-rwxrwx---+ 1 3000000 3000001     290772 Sep 27 12:38 Drafts
-rwxrwxrwx  1    1128     513 1345343427 Sep 27 12:38 Sent
drwxrwxrwx. 2    1128     513       4096 Sep 27 12:37 Archives.sbd
-rwxrwxrwx  1    1128     513      25068 Sep 27 12:32 popstate.dat

Comment 73

[WADA:World Anti-bad-Duping Agency]

(In reply to kev from comment #71)
> Sometimes, but not in every case, a rebuild causes the files to be written
> with a different user:group and unix ACL.  E.g. today's ls -lt reports:
> -rwxrwxrwx  1    1128     513      27808 Sep 27 16:00 Inbox.msf
> -rwxrwx---+ 1 3000000 3000001   20668851 Sep 27 16:00 Sent.msf
> -rwxrwxrwx  1    1128     513 2490510910 Sep 27 16:00 Sent
> -rwxrwxrwx  1    1128     513    3305947 Sep 27 15:59 Inbox

Is the "different user:group and unix ACL" surely cause/trigger of frequent rebuild-index when filesare is used?

If not(your report is about a result of rebuild-index), please open separate bug for your problem, because we are trying to find "why rebuild-index is invoked frequently if SMB1 fileshare is used for mail directory of Tb" in this bug, which is known phenomenon since Tb 3.
- Your case looks "Tb on Linux". Surely SMB1?
  Default of recent SAMBA : server max protocol = SMB2.
  This bug is for "Tb on Win-XP" and Win-XP doesn't support SMB2, so surely SMB1 case.
  If SMB2, different problem from SMB1 may exist.
- If fileshare, at least three kinds of user:group and unix ACL are relevant.
  - user:group/unix ACL of actual file at file server(Linux)
  - user:group/unix ACL like one for "shared resource" defined in fileshare server(SAMBA)
  - user:group/unix ACL at fileshare client(Linux in your case, Win-XP in this bug) 
  If user:group/unix ACL is cause of frequent rebuild-index, it may be relevant to this
  bug. But it's merely a phenomenon after rebuld-index, it's different issue.

Comment 74

[kev]

(In reply to WADA from comment #73)
> (In reply to kev from comment #71)
> > Sometimes, but not in every case, a rebuild causes the files to be written
> > with a different user:group and unix ACL.  E.g. today's ls -lt reports:
> > -rwxrwxrwx  1    1128     513      27808 Sep 27 16:00 Inbox.msf
> > -rwxrwx---+ 1 3000000 3000001   20668851 Sep 27 16:00 Sent.msf
> > -rwxrwxrwx  1    1128     513 2490510910 Sep 27 16:00 Sent
> > -rwxrwxrwx  1    1128     513    3305947 Sep 27 15:59 Inbox
> 
> Is the "different user:group and unix ACL" surely cause/trigger of frequent
> rebuild-index when filesare is used?

I'm unsure if it's a cause or a side-effect.

> If not(your report is about a result of rebuild-index), please open separate
> bug for your problem, because we are trying to find "why rebuild-index is
> invoked frequently if SMB1 fileshare is used for mail directory of Tb" in
> this bug, which is known phenomenon since Tb 3.

Yes, that's what I'm trying to find too.  I thought maybe that data would help--perhaps someone else sees these ACL/perm changes too?  Or not, and still has frequent rebuilds?

> - Your case looks "Tb on Linux". Surely SMB1?

No, Linux (CentOS) is running our Samba server (4.0.9) but the Thunderbird clients (which access their profiles on a Samba share mapped as a network drive) are on WinXPSP3.

>   Default of recent SAMBA : server max protocol = SMB2.
>   This bug is for "Tb on Win-XP" and Win-XP doesn't support SMB2, so surely
> SMB1 case.
>   If SMB2, different problem from SMB1 may exist.
> - If fileshare, at least three kinds of user:group and unix ACL are relevant.
>   - user:group/unix ACL of actual file at file server(Linux)

IIUC this is the output I was posting here, but...

>   - user:group/unix ACL like one for "shared resource" defined in fileshare
> server(SAMBA)

...I'm a bit of a Samba newb, so I'm not sure exactly which command to run to share the output of for this...is it getfacl or something else?...

>   - user:group/unix ACL at fileshare client(Linux in your case, Win-XP in
> this bug) 

...I'm also a command-line ACL newb in Windows.  What would give you helpful info here?

>   If user:group/unix ACL is cause of frequent rebuild-index, it may be
> relevant to this
>   bug. But it's merely a phenomenon after rebuld-index, it's different issue.

Yeah...not sure.  :)  Hopefully we'll figure that out soon.  Maybe someone else can confirm/deny similar behaviour on their systems.

Thanks,
Kev

Comment 75

[WADA:World Anti-bad-Duping Agency]

(In reply to kev from comment #74)
> but the Thunderbird clients (which access their profiles on a Samba share
> mapped as a network drive) are on WinXPSP3.

Oh, Tb on Win-XP with SAMBA server on Linux. i.e. surely SMB1 case.
(Recent SAMBA supports SMB2 by default, but Win-XP doesn't support SMB2.)
Pasting "ls" display on Linux(SAMBA server) is pretty confusing :-)

Do you see the "different user:group and unix ACL" at Linux/SAMBA server when newly created and accessed mail folder, or when intentional "Repair Folder"(Folder Properties" and intentional "Compact" of folder context menu?
(to force Compact execution, delete of a mail is needed)
If not, it may be a phenomenon when internal rebuild-index is invoked by some reasons.
If so, open separate bug for it, please.

Anyway, enable NSPR logging of Tb in daily use(timestamp,MSGDB:5,MsgCopyService), please, to hook "error opening db" which may be a trigger of outdated-msf condiion or broken .msf content.

Comment 76

[kev]

(In reply to WADA from comment #75)
> Do you see the "different user:group and unix ACL" at Linux/SAMBA server
> when newly created and accessed mail folder, or when intentional "Repair
> Folder"(Folder Properties" and intentional "Compact" of folder context menu?

Mine isn't having a rebuild issue nearly as much as a couple other users have been reporting, so we'll have to wait until Monday for data from them, but I can confirm that intentional compacting from the folder context menu will trigger that bug.  It also will happen with "Compact Folders" from the File menu.

> (to force Compact execution, delete of a mail is needed)

Also, this doesn't appear to be the case--

[compact Trash, then compact subfolder of Archives, each separately]
ls -lt ...
drwxrwxrwx. 2    1128     513     4096 Sep 28 11:26 Archives.sbd
-rwxrwx---+ 1 3000000 3000001     2379 Sep 28 11:25 Trash.msf
-rwxrwx---+ 1 3000000 3000001        0 Sep 28 11:25 Trash
[Compact Folders even though absolutely nothing has changed with regards to Trash since the last step]
ls -lt ...
-rwxrwx---+ 1 3000000 3000001     2433 Sep 28 11:26 Trash.msf
drwxrwxrwx. 2    1128     513     4096 Sep 28 11:26 Archives.sbd
-rwxrwx---+ 1 3000000 3000001        0 Sep 28 11:25 Trash

So apparently it's rebuilding regardless of having a delete happen or not.

> If not, it may be a phenomenon when internal rebuild-index is invoked by
> some reasons.
> If so, open separate bug for it, please.

OK, will do.  Should I put that it may be related to this one?

> Anyway, enable NSPR logging of Tb in daily
> use(timestamp,MSGDB:5,MsgCopyService), please, to hook "error opening db"
> which may be a trigger of outdated-msf condiion or broken .msf content.

Done, will report back later.

Comment 77

[kev]

(Reported other bug: https://bugzilla.mozilla.org/show_bug.cgi?id=921742 )

Comment 78

[kev]

Attachment: kevuser1debugmail.log

Comment 79

[kev]

Attachment: kevuser2debugmail.log

Comment 80

[kev]

Attachment: kevuser3debugmail.log

Comment 81

[WADA:World Anti-bad-Duping Agency]

(In reply to kev from comment #80)
> Created attachment 812145 [details]
> kevuser3debugmail.log

"error opening db 80550006" (80550006 is NS_MSG_ERROR_FOLDER_SUMMARY_MISSING == ???.msf doesn't exist) is seen for many .msf files on H: drive. If such folder is explicitly opened, internal rebuild-index will be invoked and Open log for Backup Database(C:\...\Temp\MozillaMailnews.???.msf) will be written. However, such log is not seen in your log.
i.e. .msf file of many folders is already missing in your environment.

We are not checking Tb's behavior when outdated-msf condition exists or .msf file is missing,
We are trying to find why　outdated-msf condition or missing .msf file condition is frequently occurs when fileshare is used.
Please start from non-broken set of Mbox-File/Mbox-File.msf.
  "non-broken set of Mbox-File/Mbox-File.msf" in this context is Mbox after following.
  - Delete ???.msf, Restart Tb, Click folder ??? => Rebuild-index is invoked, Termiate Tb
  - Click ??? folder, "Repair Folder", Terminate Tb
  - If outdated-msf condiion already exists, or if .msf file is already missing,
    Click folder ??? => Rebuild-index is invoked,
    Termiate Tb => Folder is closed, and MsgDatabase is physically written to .msf on HDD.

By the way, have you checked log file content before attaching to this bug by yourself? Why first log file data is garbled?

Comment 82

[Matthew Jurgens]

(In reply to WADA from comment #70)
> Do you have such message filter rules?
Yes I do.
One of these rules moves SPAM to the SPAM folder.
I have managed to get a TB log file and a procmon log file for a SPAM folder rebuild.
http://www.edcint.co.nz/LargeFiles/Completed/12.spam.TB.log
http://www.edcint.co.nz/LargeFiles/Completed/12.spam.filtered.PML.zip (procmon file filtered by path contains SPAM)
http://www.edcint.co.nz/LargeFiles/Completed/12.spam.PML.zip (Complete procmon file)

Comment 83

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #82)
> Yes I do.
> One of these rules moves SPAM to the SPAM folder.
> I have managed to get a TB log file and a procmon log file for a SPAM folder rebuild.
> http://www.edcint.co.nz/LargeFiles/Completed/12.spam.TB.log

Thanks for log data.
Quick check of MSGDB:5 log.
- "Log for outdated-msf condition of Spam folder/no Spam.msf file" is not seen in your log.
- However, internal rebuild-index looks to happen.

(1) Following log sequence is seen for Spam folder many times.
    "closing database  S:\ ...\Local Folders\Spam.msf",
    and after a short period, nsMsgDatabase::Open(S:\ ...\Local Folders\Spam.msf).
    Error doesn't look to occur.
    This is perhaps contiguous "filter move to Spam at account#A" and
    "filter move to Spam at account#B". 
(2) At 22:20:47.890000, "nsMsgDatabase::Open" is requested just after "closing database".
    (after 47msec) 
> 2013-09-29 22:20:47.890000 UTC - 0[f0f140]: closing database    S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf
> 2013-09-29 22:20:47.937000 UTC - 0[f0f140]: nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf, FALSE, 4cad710, TRUE)
(3) When nsMsgDatabase::Open is requested, currently opened MsgDatabase is listed.
    "S:\ ...\Local Folders\Spam.msf" is not listed as opened MsgDatabase.
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: 14 open DB's
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\smart mailboxes\Trash.msf - 0 hdrs in use
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\smart mailboxes\Drafts.msf - 0 hdrs in use
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\smart mailboxes\Sent.msf - 0 hdrs in use
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\smart mailboxes\Inbox.msf - 0 hdrs in use
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Search to Rebuild Folders.msf - 0 hdrs in use
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: S:\data\Mozilla Thunderbird\Matthew\mail.smartmon.com.au\Inbox.msf - 0 hdrs in use
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: S:\data\Mozilla Thunderbird\Matthew\ezyfoods.com\Inbox.msf - 0 hdrs in use
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: S:\data\Mozilla Thunderbird\Matthew\[e-mail]\Inbox.msf - 4 hdrs in use
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Inbox.msf - 65 hdrs in use
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\pop3.edcint.co.nz\Inbox.msf - 0 hdrs in use
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Unsent Messages.msf - 0 hdrs in use
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Bulk.sbd\GW.msf - 4 hdrs in use
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Bulk.sbd\eBay.msf - 3 hdrs in use
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Bulk.sbd\Nagios-Users.msf - 2 hdrs in use
(3) After 63msec from Open request, log of "Open(C:\ ... \Temp\MozillaMailnews\Spam.msf" is seen. This is perhaps log for "internal rebuild-index of Spam".(It may be log by auto-compact). Because "error opening db 80550006" is logged, "C:\ ... \Temp\MozillaMailnews\Spam.msf" was not created by Tb(there was no need to create .msf).
> 2013-09-29 22:20:48.000000 UTC - 0[f0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Spam.msf, FALSE, 4cad840, TRUE)
> 2013-09-29 22:20:48.000000 UTC - 0[f0f140]: error opening db 80550006

Why log for "Copy request" or "Copy comple notification" is not seen in your log? Did you omit MsgCopyService:5? If so, why?

Anyway, check Process Monitor log around above "close/open request within 47msec", please.

Comment 84

[WADA:World Anti-bad-Duping Agency]

Quick check of Process Monitor log.
First Write log for Spam.msf is written at ７:(=perhaps 22: in GMT)20:48.0047269.
> 477 00:00:02.8631110 7:20:48.0047269 S:\ ...\Spam.msf Write FASTIO_WRITE SUCCESS
>                                      Offset: 100,344, Length: 38 48484720 File System
This "first Write log for Spam.msf" is after following for Backup Database.
> 470 00:00:02.8622011 7:20:48.0038170 C:\ ... \Temp\MozillaMailnews\Spam.msf
>                        FASTIO_NETWORK_QUERY_OPEN NAME NOT FOUND 4848 4720 File System

Before the "first Write log for Spam.msf", "Full Read Data of .msf file" is seen multiple times.
Because "Spam.msf file is not physically writte to file", "Spam.msf data after filter move etc." is held in memory.
If "physical read of Spam.msf" is executed upon MsgDatabase open without previous "physical write of Spam.msf file to HDD", mismatch or "broken .msf" may occur.
This kind of problem, which may be relevant to "filter move", "copy message", "sent mail copy" doesn't open message folder explcitly?

Comment 85

[WADA:World Anti-bad-Duping Agency]

Quick check of Process Monitor log #2.

First log for Spam.msf is "22:19:58.796000 UTC", however, first log in Process Monitor log which is relevan to Spam/Spam.msf is "7:(== perhaps 22: in GMT)20:47.7620179".
And, there is no write log for "H: ...\Spam"(not Spam.msf). i.e. No data is added to file named Spam while Process Monitor is running.

To Matthew Jurgens(bug opener): 
Did your problem occur even though modification of Spam folder was never executed?
Did your problem occur merely by "repeaded internal open/close of Spam.msf by Tb"?

Comment 86

[Matthew Jurgens]

(In reply to WADA from comment #83)
> Did you omit MsgCopyService:5? If so, why?
No. I log with NSPR_LOG_MODULES=timestamp,MSGDB:5,MsgCopyService:5

(In reply to WADA from comment #85)
> Did your problem occur even though modification of Spam folder was never
> executed?
The only thing that ever modifies TB files is TB. My backup process only copies them (and hence may update the last access time only).

> Did your problem occur merely by "repeaded internal open/close of Spam.msf
> by Tb"?
This specific instance of the problem occurred by clicking on the Spam folder in the TB GUI. I currently have the leeway set to about 172000.

Comment 87

[kev]

(In reply to WADA from comment #81)
> (In reply to kev from comment #80)
> > Created attachment 812145 [details]
> > kevuser3debugmail.log
> 
> "error opening db 80550006" (80550006 is NS_MSG_ERROR_FOLDER_SUMMARY_MISSING
> == ???.msf doesn't exist) is seen for many .msf files on H: drive. If such
> folder is explicitly opened, internal rebuild-index will be invoked and Open
> log for Backup Database(C:\...\Temp\MozillaMailnews.???.msf) will be
> written. However, such log is not seen in your log.
> i.e. .msf file of many folders is already missing in your environment.

That's the thing.  They do actually exist.  That's why I wondered if the permissions overwrite was related to this, if maybe somehow it can't see the file because of permissions, so it makes a new one, somehow.

> 
> We are not checking Tb's behavior when outdated-msf condition exists or .msf
> file is missing,
> We are trying to find why　outdated-msf condition or missing .msf file
> condition is frequently occurs when fileshare is used.

I'm aware of that.

> Please start from non-broken set of Mbox-File/Mbox-File.msf.
>   "non-broken set of Mbox-File/Mbox-File.msf" in this context is Mbox after
> following.
>   - Delete ???.msf, Restart Tb, Click folder ??? => Rebuild-index is
> invoked, Termiate Tb
>   - Click ??? folder, "Repair Folder", Terminate Tb
>   - If outdated-msf condiion already exists, or if .msf file is already
> missing,
>     Click folder ??? => Rebuild-index is invoked,
>     Termiate Tb => Folder is closed, and MsgDatabase is physically written
> to .msf on HDD.

We had done this, with the exception of actually deleting the .msf.

> 
> By the way, have you checked log file content before attaching to this bug
> by yourself? Why first log file data is garbled?

I'm not sure--I hadn't noticed until you mentioned it, because I didn't have word wrap on in my text editor, so it was just one "line", which didn't grab my attention.

Comment 88

[Matthew Jurgens]

(In reply to WADA from comment #84)
> This kind of problem, which may be relevant to "filter move", "copy
> message", "sent mail copy" doesn't open message folder explcitly?
I have filter move rules which move messages to Spam and Trash. These are very common folders for rebuilds. However, Sent folder also rebuilds frequently. I have no rules myself to move items to the Sent folder. I guess the saving of emails I send to the Sent folder my have the same process though as a filter move rule.

Comment 89

[WADA:World Anti-bad-Duping Agency]

(In reply to kev from comment #78)
> kevuser1debugmail.log
Log content is following line only. 
> Pretty long grabage + 2013-09-30 17:36:00.355000 UTC - 0[d0f140]: closing database H:\Thunderbird\Mail\pop.brantaero.com\Sent.msf
(In reply to kev from comment #79)
> kevuser2debugmail.log
Log content is following .
> Pretty long grabage + 2013-09-30 16:59:12.265000 UTC - 0[d0f140]: closing database    H:\Thunderbird\Mail\smart mailboxes\Sent.msf
> 2013-09-30 17:10:27.799000 UTC - 0[d0f140]: CopyMessages
>                   |
> 2013-09-30 17:35:46.926000 UTC - 0[d0f140]: closing database    H:\Thunderbird\Mail\pop.registeredsite.com\Sent.msf
(In reply to kev from comment #80)
> kevuser3debugmail.log
Log content is following.
> 2013-09-30 12:21:40.481000 UTC - 0[e0f140]: nsMsgDatabase::Open(H:\Mozilla Data\Mail\pop.registeredsite.com\Inbox.msf, FALSE, 58df420, TRUE)
>                   |
> 2013-09-30 17:41:41.674000 UTC - 0[e0f140]: request cb33b00 Clearing OK request - src mailbox://[e-mail]/Inbox dest mailbox://nobody@Local%20Folders/Trash numItems 3 type=0

kev, how did you generate the three log files?
Please attach entire NSPR log for "from start of Tb till termination of Tb".
To avoid garbled data, delete log file before "start of Tb with logging enabled".

(In reply to kev from comment #87)
> > Please start from non-broken set of Mbox-File/Mbox-File.msf.
> >   "non-broken set of Mbox-File/Mbox-File.msf" in this context is Mbox after
> > following.
> > (snip)
> We had done this, with the exception of actually deleting the .msf.

As for Sent folder, you look to have done it just after restart of Tb.

kevuser3debugmail.log does look "log from start of Tb", because it starts with following.
> 2013-09-30 12:21:40.481000 UTC - 0[e0f140]: nsMsgDatabase::Open(H:\Mozilla Data\Mail\pop.registeredsite.com\Inbox.msf, FALSE, 58df420, TRUE)
> 2013-09-30 12:21:40.481000 UTC - 0[e0f140]: 0 open DB's
However, it doesn't look "entire log", because following is log which is relevant to Sent folder,
(by folder click, rebuild-index is done, and after a while, one sent mail is copied to Sent)
> 2013-09-30 12:21:48.186000 UTC - 0[e0f140]: nsMsgDatabase::Open(H:\Mozilla Data\Mail\pop.registeredsite.com\Sent.msf, FALSE, edd550, TRUE)
> 2013-09-30 12:21:49.642000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\DOCUME~1\Pat\LOCALS~1\Temp\MozillaMailnews\Sent.msf, FALSE, 58ded00, TRUE)
> 2013-09-30 12:21:49.642000 UTC - 0[e0f140]: error opening db 80550006
>
> 2013-09-30 12:21:49.657000 UTC - 0[e0f140]: closing database    H:\Mozilla Data\Mail\pop.registeredsite.com\Sent.msf
> 2013-09-30 12:21:49.657000 UTC - 0[e0f140]: closing database    C:\DOCUME~1\Pat\LOCALS~1\Temp\MozillaMailnews\Sent.msf
> 2013-09-30 12:21:50.487000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\DOCUME~1\Pat\LOCALS~1\Temp\MozillaMailnews\Sent.msf, FALSE, edd550, TRUE)
> 2013-09-30 12:21:50.973000 UTC - 0[e0f140]: nsMsgDatabase::Open(H:\Mozilla Data\Mail\pop.registeredsite.com\Sent.msf, TRUE, 58ded00, TRUE)
>
> 2013-09-30 12:21:50.973000 UTC - 0[e0f140]: 4 open DB's
> 2013-09-30 12:21:50.973000 UTC - 0[e0f140]: C:\DOCUME~1\Pat\LOCALS~1\Temp\MozillaMailnews\Sent.msf - 0 hdrs in use
>
> 2013-09-30 12:21:59.789000 UTC - 0[e0f140]: 8 open DB's
> 2013-09-30 12:21:59.789000 UTC - 0[e0f140]: C:\DOCUME~1\Pat\LOCALS~1\Temp\MozillaMailnews\Sent.msf - 6 hdrs in use
> 2013-09-30 12:21:59.789000 UTC - 0[e0f140]: H:\Mozilla Data\Mail\pop.registeredsite.com\Sent.msf - 11 hdrs in use
>
> 2013-09-30 12:24:32.637000 UTC - 0[e0f140]: closing database    C:\DOCUME~1\Pat\LOCALS~1\Temp\MozillaMailnews\Sent.msf
>
> 2013-09-30 12:25:44.840000 UTC - 0[e0f140]: request 43abec0 DoCopy - src  dest mailbox://[e-mail]/Sent numItems 0 type=1
> 2013-09-30 12:25:44.887000 UTC - 0[e0f140]: NotifyCompletion - src  dest mailbox://[e-mail]/Sent
> 2013-09-30 12:25:44.887000 UTC - 0[e0f140]: request 43abec0 Clearing OK request - src  dest mailbox://[e-mail]/Sent numItems 0 type=1
>
> 2013-09-30 12:25:51.302000 UTC - 0[e0f140]: CopyMessages
> 2013-09-30 12:25:51.302000 UTC - 0[e0f140]: request 5d0c3c0 CopyMessages request - src mailbox://[e-mail]/Inbox dest mailbox://[e-mail]/Pat/Avionics%20Quotes/Customer%20%20information numItems 0 type=0
> 2013-09-30 12:25:51.302000 UTC - 0[e0f140]: request 5d0c3c0 DoCopy - src mailbox://[e-mail]/Inbox dest mailbox://[e-mail]/Pat/Avionics%20Quotes/Customer%20%20information numItems 1 type=0
> 2013-09-30 12:25:51.302000 UTC - 0[e0f140]: nsMsgDatabase::Open(H:\Mozilla Data\Mail\pop.registeredsite.com\Pat.sbd\Avionics Quotes.sbd\Customer  information.msf, FALSE, 9414bd0, TRUE)
>
> 2013-09-30 12:25:51.834000 UTC - 0[e0f140]: 4 open DB's
> 2013-09-30 12:25:51.834000 UTC - 0[e0f140]: H:\Mozilla Data\Mail\pop.registeredsite.com\Sent.msf - 1 hdrs in use
>
> 2013-09-30 12:26:53.832000 UTC - 0[e0f140]: closing database    H:\Mozilla Data\Mail\pop.registeredsite.com\Sent.msf
>
> 2013-09-30 12:26:54.537000 UTC - 0[e0f140]: request 4333b40 DoCopy - src  dest mailbox://[e-mail]/Sent numItems 0 type=1
> 2013-09-30 12:26:54.552000 UTC - 0[e0f140]: nsMsgDatabase::Open(H:\Mozilla Data\Mail\pop.registeredsite.com\Sent.msf, FALSE, 58ded00, TRUE)
>
> 2013-09-30 12:26:55.194000 UTC - 0[e0f140]: NotifyCompletion - src  dest mailbox://[e-mail]/Sent
> 2013-09-30 12:26:55.194000 UTC - 0[e0f140]: request 4333b40 Clearing OK request - src  dest mailbox://[e-mail]/Sent numItems 0 type=1
> 2013-09-30 12:26:58.018000 UTC - 0[e0f140]: nsMsgDatabase::Open(H:\Mozilla Data\Mail\pop.registeredsite.com\Inbox.msf, FALSE, 7296da0, TRUE)
>
> 2013-09-30 12:26:58.034000 UTC - 0[e0f140]: 3 open DB's
> 2013-09-30 12:26:58.034000 UTC - 0[e0f140]: H:\Mozilla Data\Mail\pop.registeredsite.com\Sent.msf - 1 hdrs in use
>
> 2013-09-30 12:27:17.004000 UTC - 0[e0f140]: closing database    H:\Mozilla Data\Mail\pop.registeredsite.com\Sent.msf
and because last log is following for "Delete 3 mails in Inbox".
> 2013-09-30 17:41:41.674000 UTC - 0[e0f140]: NotifyCompletion -
>                     src mailbox://[e-mail]/Inbox
>                     dest mailbox://nobody@Local%20Folders/Trash
> 2013-09-30 17:41:41.674000 UTC - 0[e0f140]: request cb33b00 Clearing OK request -
>                     src mailbox://[e-mail]/Inbox
>                     dest mailbox://nobody@Local%20Folders/Trash numItems 3 type=0

Why no "error opening db" for Sent.msf just after restart of Tb, even though Rebuld-Index is invoked? It's manual/intentional "Repair Folder"?

If so, no problem. But "all log after it till next outdated-msf condition or next msf corruption" is needed for problem analysis, isn't it?

If it's "internal rebuild-index upon folder click", "no error opening db on Sent.msf" may mean "Sent.msf corruption by Tb". If so, it may be relevant to "numItems 0 when type=1 in Copy request". However, if so, I think "rebuild-index upon folder click" should always occur on Sent folder after restart of Tb when "copy to Sent" happened during previous Tb run.

Comment 90

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #82)
> I have managed to get a TB log file and a procmon log file for a SPAM folder rebuild.
> http://www.edcint.co.nz/LargeFiles/Completed/12.spam.TB.log
(In reply to Matthew Jurgens from comment #86)
> (In reply to WADA from comment #83)
> > Did you omit MsgCopyService:5? If so, why?
> No. I log with NSPR_LOG_MODULES=timestamp,MSGDB:5,MsgCopyService:5

If so, nothing was copied to any folder using MsgCopyService while you were getting Tb log, if log file you attached is entire log data from restart of Tb till termination of Tb.

Following is extracted log lines which has \Spam.msf and \Search to Rebuild Folders.msf, "error opening db" line, and "closing" log without file name in log line.
> 2013-09-29 22:19:39.734000 UTC - 0[f0f140]: nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Search to Rebuild Folders.msf, FALSE, b8f2110, TRUE)
>
> 2013-09-29 22:19:58.796000 UTC - 0[f0f140]: nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf, FALSE, 9760700, TRUE)
>
> 2013-09-29 22:19:58.859000 UTC - 0[f0f140]: 12 open DB's (No \Spam.msf in opened DB list)
>
> 2013-09-29 22:20:05.546000 UTC - 0[f0f140]: closing database    S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf
>
> 2013-09-29 22:20:09.484000 UTC - 0[f0f140]: nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf, FALSE, 4cab960, TRUE)
>
> 2013-09-29 22:20:09.500000 UTC - 0[f0f140]: 11 open DB's (No \Spam.msf in opened DB list)
>
> 2013-09-29 22:20:09.500000 UTC - 0[f0f140]: closing database    S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf
> 2013-09-29 22:20:13.968000 UTC - 0[f0f140]: nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf, FALSE, 4cab830, TRUE)
>
> 2013-09-29 22:20:14.000000 UTC - 0[f0f140]: 11 open DB's (No \Spam.msf in opened DB list)
>
> 2013-09-29 22:20:14.000000 UTC - 0[f0f140]: closing database    S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf
>
> 2013-09-29 22:20:47.750000 UTC - 0[f0f140]: nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf, FALSE, 4cad710, TRUE)
>
> 2013-09-29 22:20:47.812000 UTC - 0[f0f140]: 14 open DB's (No \Spam.msf in opened DB list)
>
> 2013-09-29 22:20:47.890000 UTC - 0[f0f140]: closing database    S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf
> 2013-09-29 22:20:47.937000 UTC - 0[f0f140]: nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf, FALSE, 4cad710, TRUE)
>
> 2013-09-29 22:20:47.953000 UTC - 0[f0f140]: 14 open DB's (No \Spam.msf in opened DB list)
>
> 2013-09-29 22:20:48.000000 UTC - 0[f0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Spam.msf, FALSE, 4cad840, TRUE)
> 2013-09-29 22:20:48.000000 UTC - 0[f0f140]: error opening db 80550006
>
> 2013-09-29 22:20:48.000000 UTC - 0[f0f140]: 14 open DB's (No \Spam.msf in opened DB list)
>
> 2013-09-29 22:20:48.000000 UTC - 0[f0f140]: closing database    
> 2013-09-29 22:20:48.000000 UTC - 0[f0f140]: closing database    S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf
> 2013-09-29 22:20:48.000000 UTC - 0[f0f140]: closing database    C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Spam.msf
> 2013-09-29 22:20:48.046000 UTC - 0[f0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Spam.msf, FALSE, 4cad710, TRUE)
>
> 2013-09-29 22:20:48.125000 UTC - 0[f0f140]: 14 open DB's (No \Spam.msf in opened DB list)
>
> 2013-09-29 22:20:48.156000 UTC - 0[f0f140]: nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf, TRUE, 4cad840, TRUE)
>
> 2013-09-29 22:20:48.218000 UTC - 0[f0f140]: 14 open DB's (No \Spam.msf in opened DB list)e
> 2013-09-29 22:20:48.890000 UTC - 0[f0f140]: closing database    C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Spam.msf
> 2013-09-29 22:20:52.359000 UTC - 0[f0f140]: closing database    S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf
>
> 2013-09-29 22:20:52.625000 UTC - 0[f0f140]: 14 open DB's (No \Spam.msf in opened DB list)
>
> 2013-09-29 22:20:52.984000 UTC - 0[f0f140]: closing database    S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Search to Rebuild Folders.msf

Is "\Search to Rebuild Folders.msf" virtual folder to check \Spam, \Trash, \Sent etc. to see some errors occurred on folder or not? If so, you kept the folder open at folder pane? Repeated open of S:\...\Local Folders\Spam.msf is by opening "\Search to Rebuild Folders"?

If "outdated-msf condition"(different file size because pretty large timestamp_leeway is used) or "missing .msf file", I think "error opening db 80550005" or "error opening db 80550006" should be logged for S:\...\Local Folders\Spam.msf before rebuild-index is invoked upon folder click.
If it's right, I can't imagine other than followings.
(1) S:\...\Local Folders\Spam.msf is already corrupted before restart of Tb.
(2) If S:\...\Local Folders\Spam.msf is healthy just after restart of Tb, Junk move or someone doesn't use MsgCopyService, so log by MsgCopyService:5 is not written. If this case, "No \Spam.msf in opened DB list" may be relevant to it.
(3) If S:\...\Local Folders\Spam.msf is healthy just after restart of Tb, S:\...\Local Folders\Spam.msf is corrupted merely by repeated open/close of S:\...\Local Folders\Spam.msf. If this case, "closing log without file name" may be relevant to problem, and "immediate open just after close" may be relevant to it. "No \Spam.msf in opened DB list" may also be relevant.

Comment 91

[WADA:World Anti-bad-Duping Agency]

To Matthew Jurgens(bug opener) and kev:
Please surely rule out "(1) X:\...\Mbox.msf is already corrupted before restart of Tb." case, please.
It can be achieved by;
(1-1) With Tb logging enabled, restart Tb, click Mbox at folder pane and view a mail, or execute Repair Folder of the Mbox intentionally and view a mail, then terminate Tb.
(1-2) Check log and confirm that internal rebuid-index due to something wrong was not invoked.
(2-1) Restart Tb with logging enabled, click Mbox at folder pane and view a mail, click other folder at folder pane => Mbox.msf will be closed sooner or later.
(2-2) After a while(5 to 10 minutes), copy Tb's log file to work file, and check work file content. Except data still held in buffer, all data is copied to work file. Confirm that internal rebuild-index was not invoked by click of Mbox at folder pane, and confirm thet Mbox.msf was closed.
(3) Keep Tb running. If "outdated-msf condition" or "missing .msf" is detected(for example, exception in Error Console), or internal rebuild-index is invoked, terminate Tb and check log file content.
Note:
For problem analysis, "Process Monitor log from (2-1) till end of (3)" is usually required. However, Process Monitor log will become pretty huge. So, it's usually very hard, unless huge free disk space is available.

Comment 92

[Matthew Jurgens]

> Is "\Search to Rebuild Folders.msf" virtual folder to check \Spam, \Trash,
> \Sent etc. to see some errors occurred on folder or not? If so, you kept the
> folder open at folder pane? Repeated open of S:\...\Local Folders\Spam.msf
> is by opening "\Search to Rebuild Folders"?

The "Search to Rebuild Folders.msf" is a virtual folder that shows mails from all folders that is less than 1 day old. I have done this as a simple way to quickly open all folders, which rebuilds them as needed. This is significantly easier than clicking each folder manually.

Comment 93

[Matthew Jurgens]

(In reply to WADA from comment #91)

> (1-2) Check log and confirm that internal rebuid-index due to something
> wrong was not invoked.
> (2-2) After a while(5 to 10 minutes), copy Tb's log file to work file, and
> check work file content. Except data still held in buffer, all data is
> copied to work file. Confirm that internal rebuild-index was not invoked by
> click of Mbox at folder pane, and confirm thet Mbox.msf was closed.

Specifically, how do I 
1) "confirm that internal rebuid-index"
2) "confirm thet Mbox.msf was closed"

And when I "check work file content" <--- what exactly am I looking for?

Comment 94

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #93)
> Specifically, how do I 
> 1) "confirm that internal rebuid-index"
> 2) "confirm thet Mbox.msf was closed"

Read thru this bug well.
I already repeatedly explained that "open of C:...\Temp\MozillaMealnews\MboxName.msf" is logged when rebuild-index is executed.
If you don't do explicit "Repair Folder", such log is evidence that internal rebuld-index was invoked.
And, "closing database ..." is logged by MSGDB:5. You never read NSPR log by yourself?

> And when I "check work file content" <--- what exactly am I looking for?
Needless to say, we are trying to find "when cause of internal rebuild-index was generated while Tb is running". So, needless to say, at least following is needed.
1. Check when rebuild-index was invoked upon mail folder click, even thouh you didn't request Repair Folder.
2. Check all logs relevant to the mail folder, and try to find "when something wrong happened" by　ｃhecking all logs in log file which is relevant to the mail folder.

Comment 95

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #92)
> The "Search to Rebuild Folders.msf" is a virtual folder that shows mails
> from all folders that is less than 1 day old. I have done this as a simple
> way to quickly open all folders, which rebuilds them as needed.
> This is significantly easier than clicking each folder manually.

As I previously wrote, I couldn't observe "internal rebuild-index by simple virtual folder open when outdated-msf condition, which is intentionally created, exists". I could see only "exception in Error Console with 80550005 or 80550006" by virtual folder open.
As I previously wrote, I could consistently observe "internal rebuild-index when intentionally-created outdated-msf condition exists" only when "folder click"(intentional folder open) and "showing Folder Properties".

Did you actually observe "internal rebuild-index when outdated-msf condition exists" consistently, simply by "virtual folder open"?

Comment 96

[Matthew Jurgens]

So here is what I did today.
8am:
1) Opened TB and used my virtual folder "Search to Rebuild Folders" to open all folders and make sure that they were all in the correct state.
2) Shutdown TB
3) Disable backups of TB

No activity on TB files. Leeway set to 172000 seconds (ie 2 days)

6pm (10 hours later):
4) Checked that no backups were taken today - confirmed.
5) Opened TB with logging enabled.
6) Waited for new mail to download. No problems so far.
7) Started Procmon
8) used my virtual folder "Search to Rebuild Folders" to see if any folders would be rebuilt
9) Watched as the folder called Nagios-Users was rebuilt (it was the only one).
10) Clicked on Nagios-Users folder to see if there was new mail in it which might have been moved there by a rule. There was no new mail. I did see a new message the "1 mail was deleted". It might have been from Nagios-Users since Nagios-Users has a retention rule of something like 180 days.
11) Closed TB. Stopped Procmon
12) Opened the Nagios and located the follow error lines (timestamps are UTC and we are currently UTC+11):
2013-10-09 07:00:40.255000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Nagios-Users.msf, FALSE, 9609cf0, TRUE)
2013-10-09 07:00:40.255000 UTC - 0[e0f140]: error opening db 80550006
13) Opened the Procmon file, filtered it by Path contains Nagios-Users and found the following lines at the same timestamp:
6:00:40.2555621 PM	thunderbird.exe	5636	QueryDirectory	C:\Documents and Settings\mjurgens\Local Settings\Temp\MozillaMailnews\Nagios-Users.msf	NO SUCH FILE	Filter: Nagios-Users.msf
6:00:40.2820832 PM	thunderbird.exe	5636	QueryOpen	C:\Documents and Settings\mjurgens\Local Settings\Temp\MozillaMailnews\Nagios-Users.msf	NAME NOT FOUND	
6:00:40.2822082 PM	thunderbird.exe	5636	QueryOpen	C:\Documents and Settings\mjurgens\Local Settings\Temp\MozillaMailnews\Nagios-Users.msf	NAME NOT FOUND	
6:00:40.2823428 PM	thunderbird.exe	5636	QueryOpen	C:\Documents and Settings\mjurgens\Local Settings\Temp\MozillaMailnews\Nagios-Users.msf	NAME NOT FOUND	


TB rebuilds seems to break the folders all by itself (It goes looking for a file that does not exist) and then rebuilds them when I use a virtual folder to open them.
So what now?

Comment 97

[Matthew Jurgens]

(In reply to Matthew Jurgens from comment #96)
The log files and procmon files for Comment 96 are
http://www.edcint.co.nz/LargeFiles/Completed/13.Nagios-Users.TB.log.zip
http://www.edcint.co.nz/LargeFiles/Completed/13.Nagios-Users.Filtered.PML.zip
http://www.edcint.co.nz/LargeFiles/Completed/13.Nagios-Users.PML.zip

Comment 98

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #96)
Because log for "outdated-msf" nor ".msf is missing" is not seen, it may be ".msf file corruption".
And it maight happen when.
> (a) During "2) Shutdown TB"
> (b) Between "5) Opened TB with logging enabled" and "7) Started Procmon"
> (c) Between "5) Opened TB with logging enabled" and "8) used my virtual folder Search to Rebuild Folders".
> (d) By "8) used my virtual folder Search to Rebuild Folders"
I'm now suspecting (a), because "update action on Nagios-Users after restart" is "one message purge" only, or problem by "message purge".
And, I now guess that it's not SMB1 relevant issue. I guess timinig relevant issue which can occur on local HDD file. I guess that it's merely "timing whole is very large" because file writing takes pretty long when fileshare is used and file open/close takes relatively long when fileshare is used.

Matthew Jurgens, what do you think?

Do you enable "Retention Policy" of many folders especially for "folders rebuild-infex is frequently seen"?

By the way, MsgPurge is logged by MsgPurge:5. IIRC, volume of log by MsgPurge:5 is perhaps larger than MSGDB:5. Please be careful when you enable MsgPurge:5. Check log volume before enable it in your daily use.

Comment 99

[Matthew Jurgens]

(In reply to WADA from comment #98)
> Matthew Jurgens, what do you think?
I agree it could be a timing issue (previously suggested in comment #33).

> Do you enable "Retention Policy" of many folders especially for "folders
> rebuild-infex is frequently seen"?
Only some other folders I use and that are rebuilt also use retention policies however I believe the common factor is that there is some rule (internal TB or user defined) that moves mail to a folder. These target folders then seem to be likely candidates for a rebuild.

Comment 100

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #96)
> 8) used my virtual folder "Search to Rebuild Folders" to see if any folders would be rebuilt
> 9) Watched as the folder called Nagios-Users was rebuilt (it was the only one).
How did you confirm "Nagios-Users was rebuilt(it was the only one)"?

Please note that "if rebuild-index is invoked on Mbox, C:\...\Temp\MozillaMailnews/Mbox.msf is used by Tb" is true but "any log of C:\...\Temp\MozillaMailnews/Mbox.msf is evidence of 'rebuild-index was invoked'" is perhaps never true. Please note that we don't know full spec and behavior of Tb on Backup Database yet...

Comment 101

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #99)
> > Do you enable "Retention Policy" of many folders especially for "folders
> > rebuild-infex is frequently seen"?
> Only some other folders I use and that are rebuilt also use retention
> policies however I believe the common factor is that there is some rule
> (internal TB or user defined) that moves mail to a folder. These target
> folders then seem to be likely candidates for a rebuild.

If so, "cause of rebuild-index of Nagios-Users" was generated during previous Tb run, or during termination of previous Tb run...
What kind of data is needed for problem analysis? Who can analize data for problem determination?

Comment 102

[Matthew Jurgens]

(In reply to WADA from comment #100)
> How did you confirm "Nagios-Users was rebuilt(it was the only one)"?
I saw it in the status bar (like in the attached video https://bugzilla.mozilla.org/attachment.cgi?id=792120 that I previously attached)

(In reply to WADA from comment #101)
> If so, "cause of rebuild-index of Nagios-Users" was generated during
> previous Tb run, or during termination of previous Tb run...
I don't know

> What kind of data is needed for problem analysis? Who can analize data for
> problem determination?
I don't know. I was hoping you could :)

Comment 103

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #96)
> The log files and procmon files for Comment 96 are
> http://www.edcint.co.nz/LargeFiles/Completed/13.Nagios-Users.TB.log.zip

Difference between following two mail folders, during '(b) Between "5) Opened TB with logging enabled" and "7) Started Procmon"'(i.e. no Process Monitor log is available).
> S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Bulk.sbd\Nagios-Users.msf
> S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Bulk.sbd\MythTV.msf
Nagios-Users.msf :
  After Open request including first open request after restart of Tb,
  Nagios-Users.msf is not shown in list of "nn open DB's",
  and is immediately closed.
  This is repeated frequently, until rebuild-index is executed.
MythTV.msf :
  After Open request at 07:00:32.099000, close is done at 07:00:40.708000.
  MythTV.msf is always shown in list of "nn open DB's" before close,
  and "S:\...\Bulk.sbd\MythTV.msf - 0 hdrs in use" is logged.

If both Nagios-Users and MythTV are search target folder of Virtual Folder, and if Retention Policy is set for both mail folders, I think Nagios-Users.msf is already broken before restart of Tb.

Comment 104

[WADA:World Anti-bad-Duping Agency]

FYI.
When restart of Tb, Tb opens last selected folder. So, if you click "edcint.co.nz\Search to Rebuild Folders" at folder pane just before termination of Tb, "Search to Rebuild Folders" folder is always opened just after restart of Tb. If you start Process Monitor just before restart of Tb, you can get Process Monitor log for "from initial of restart till an event".

Comment 105

[WADA:World Anti-bad-Duping Agency]

To Matthew Jurgens(bug opener) and kev:

Problem doesn't look outdated-msf condition related issue which is caused by SMB1 use.
In order to avoid misleading, please use default timestamp_leeway value in your daily use and testing.

By the way, as I wrote before, one of biggest difference between "Tb 1/2" and "Tb 3 or later" is;
 Tb 1/2        : .msf file is never written/closed until termination of Tb.
 Tb 3 or later : after mail folder close, .msf file is written and closed sooner or later.
                 time of "sooner or later" depends on Tb version and hidden prefs setting.
I guess this is relevant to problem, if this bug is new problem started from Tb 3. Problem may be that ".msf file writing and close process after mail folder close" is killed or interfered by termination process. If this kind of problem, "why file share only problem" can be explained, because "entire .msf file writing" takes pretty long when file share is used, and "why problem won't occur on data file(file named Spam instead of Spam.msf) even when file share is used" can be explained too.

Comment 106

[kev]

Attachment: kevdebugmail.log - latest from freshly rebuilt .msfs, no timestamp setting

Comment 107

[kev]

1) deleted all .msf
2) moved my mail back to fileshare from local, switched local profiles.ini to point to mapped H: drive on fileshare
3) timestamp_leeway checked--not defined in prefs.js so I must not have set it on mine but only on one of my users' before
4) launched TB, did File->Compact Folders and waited until "Done Compacting"

At this point, my dir looks like this:

> ls -l -R /home/srv/users/Kev/Thunderbird/Mail/pop.brantaero.com/
/home/srv/users/Kev/Thunderbird/Mail/pop.brantaero.com/:
total 95356
-rwxrwx---+ 1 3000000 3000001        0 Oct 18  2012 Archives
-rwxrwx---+ 1 3000000 3000001        0 Oct 10 09:25 Archives.msf
drwxrwx---+ 2 3000000 3000001     4096 Oct 10 09:27 Archives.sbd
-rwxrwx---+ 1 3000000 3000001        0 Oct  8 10:54 Drafts
-rwxrwx---+ 1 3000000 3000001        0 Oct 10 09:25 Drafts.msf
-rwxrwx---+ 1 3000000 3000001       80 Sep 26 18:06 filterlog.html
-rwxrwx---+ 1 3000000 3000001  1159658 Oct 10 09:26 Inbox
-rwxrwx---+ 1 3000000 3000001     6304 Oct 10 09:26 Inbox.msf
drwxrwx---+ 3 3000000 3000001     4096 Oct 10 09:25 Inbox.sbd
-rwxrwx---+ 1 3000000 3000001       28 Oct  2  2012 msgFilterRules.dat
-rwxrwx---+ 1 3000000 3000001    19856 Oct 10 09:27 popstate.dat
-rwxrwx---+ 1 3000000 3000001 96351100 Oct  9 20:00 Sent
-rwxrwx---+ 1 3000000 3000001        0 Oct 10 09:25 Sent.msf
-rwxrwx---+ 1 3000000 3000001        0 Mar 25  2010 Templates
-rwxrwx---+ 1 3000000 3000001        0 Oct 10 09:25 Templates.msf
-rwxrwx---+ 1 3000000 3000001    39127 Oct  9 14:54 Trash
-rwxrwx---+ 1 3000000 3000001        0 Oct 10 09:25 Trash.msf

/home/srv/users/Kev/Thunderbird/Mail/pop.brantaero.com/Archives.sbd:
total 558032
-rwxrwx---+ 1 3000000 3000001   8478775 Oct 18  2012 2005
-rwxrwx---+ 1 3000000 3000001         0 Oct 10 09:25 2005.msf
-rwxrwx---+ 1 3000000 3000001  53355647 Dec 11  2012 2006
-rwxrwx---+ 1 3000000 3000001         0 Oct 10 09:25 2006.msf
-rwxrwx---+ 1 3000000 3000001  49466882 Dec 11  2012 2007
-rwxrwx---+ 1 3000000 3000001         0 Oct 10 09:25 2007.msf
-rwxrwx---+ 1 3000000 3000001 126699657 Oct 26  2012 2008
-rwxrwx---+ 1 3000000 3000001         0 Oct 10 09:25 2008.msf
-rwxrwx---+ 1 3000000 3000001  84991858 Oct 18  2012 2009
-rwxrwx---+ 1 3000000 3000001         0 Oct 10 09:25 2009.msf
-rwxrwx---+ 1 3000000 3000001 120416483 Oct 18  2012 2010
-rwxrwx---+ 1 3000000 3000001         0 Oct 10 09:25 2010.msf
-rwxrwx---+ 1 3000000 3000001    661529 Oct 18  2012 2011
-rwxrwx---+ 1 3000000 3000001         0 Oct 10 09:25 2011.msf
-rwxrwx---+ 1 3000000 3000001  11058695 Jan 15  2013 2012
-rwxrwx---+ 1 3000000 3000001         0 Oct 10 09:25 2012.msf
-rwxrwx---+ 1 3000000 3000001 115324905 Oct 10 09:27 2013
-rwxrwx---+ 1 3000000 3000001    910732 Oct 10 09:27 2013.msf

/home/srv/users/Kev/Thunderbird/Mail/pop.brantaero.com/Inbox.sbd:
total 4088
-rwxrwx---+ 1 3000000 3000001 4169895 Oct  3 10:17 Todo
-rwxrwx---+ 1 3000000 3000001       0 Oct 10 09:25 Todo.msf
drwxrwx---+ 2 3000000 3000001    4096 Oct 10 09:25 Todo.sbd

/home/srv/users/Kev/Thunderbird/Mail/pop.brantaero.com/Inbox.sbd/Todo.sbd:
total 36
-rwxrwx---+ 1 3000000 3000001 30025 Sep 16 09:19 FBO updates
-rwxrwx---+ 1 3000000 3000001     0 Oct 10 09:25 FBO updates.msf


5) quit TB and checked that thunderbird.exe process was no longer running using Windows Task Manager with "Show From All Users" checked
6) deleted kevdebugmail.log
7) launched my .cmd file that contains this:

set NSPR_LOG_MODULES=timestamp,MSGDB:5,MsgCopyService:5
set NSPR_LOG_FILE=C:\kevdebugmail.log
start "" "%ProgramFiles%\Mozilla Thunderbird\thunderbird.exe"

8) Inbox was showing, mail auto-checked immediately.  I then clicked the folder FBO Updates.  Instead of showing the mail immediately, there was a brief second (over remote desktop, so it's hard to say whether I saw everything or not) where the "folder page" showed instead of a list of mail, after which it showed the list of mail.  I mention this because one of my users was having this folder page show for a while during summary rebuilds after our switch to Samba.  So, it might've rebuilt, very quickly, because the folder is small.  Indeed:

> ls -l -R /home/srv/users/Kev/Thunderbird/Mail/pop.brantaero.com/Inbox.sbd/Todo.sbd
/home/srv/users/Kev/Thunderbird/Mail/pop.brantaero.com/Inbox.sbd/Todo.sbd:
total 44
-rwxrwx---+ 1 3000000 3000001 30025 Sep 16 09:19 FBO updates
-rwxrwx---+ 1 3000000 3000001  3941 Oct 10 09:30 FBO updates.msf

9) I then clicked the "Sent" folder.  This time, folder page, and definitely rebuilding the summary, it says so in the status bar.  Once the mail came up, I quit TB, and again confirmed thunderbird.exe is no longer running.  It's from this point that I'm attaching the latest kevdebugmail.log.

Comment 108

[kev]

Re: filters, if I go to Tools -> Message Filters, none exist in my main mail folder nor Local Folders, which are the only two options in the drop-down menu.

Comment 109

[kev]

Oh and BTW, this is on SerNet Samba 4.0.10, just recently released.

Comment 110

[Matthew Jurgens]

(In reply to WADA from comment #103)

Now using the following TB logging:
SET NSPR_LOG_MODULES=timestamp,MSGDB:5,MsgCopyService:5,MsgPurge:5

So here is what I did in the last 24 hours.
8am Friday:
1) Disable backups of TB
2) Opened TB with logging enabled and used my virtual folder "Search to Rebuild Folders" to open all folders and make sure that they were all in the correct state.
3) Shutdown TB
4) Repeat 1 and 2 multiple times
5) Start Procmon
6) Opened TB with logging enabled and used my virtual folder "Search to Rebuild Folders" to open all folders and make sure that they were all in the correct state.
7) Shutdown TB
8) Stop Procmon

No activity on TB files. Leeway set to 4000 seconds

9am Saturday (approx 24 hours later):
9) Checked that no backups were taken - confirmed.
10) Started Procmon. Opened TB with logging enabled.
11) Waited for new mail to download. No problems so far.
12) used my virtual folder "Search to Rebuild Folders" to see if any folders would be rebuilt
13) Watched as the folder called Spam and Sent were rebuilt.
14) Closed TB. Stopped Procmon
15) Opened the TB Log just generated and located the 80550006 errors to match the 2 folders rebuilt
16) Opened the Procmon file, filtered it by Path contains Sent and found the matching NO SUCH FILE error, filtered it by Path contains Spam and found the matching NO SUCH FILE error

I have left the Procmon files unfiltered so you can look at other things that we might not have thought of yet.
The TB log and unfiltered Procmon File from steps 5 and 6 are at:
http://www.edcint.co.nz/LargeFiles/Completed/14.Pre.TB.zip

The TB log and unfiltered Procmon File from step 10 are at:
http://www.edcint.co.nz/LargeFiles/Completed/14.Sent.Spam.TB.zip

Comment 111

[WADA:World Anti-bad-Duping Agency]

Spam == S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf ?

For Local Folders\Spam:

In 14.Sent.Spam.TB.log, (a) log sequence of "just after open Local Folders\Spam.msf, immediately close Local Folders\Spam.msf" is seen, and (b) second log sequence of "just after open Local Folders\Spam.msf, immediately close Local Folders\Spam.msf" is seen and (c) following is logged.
> 2013-10-11 22:22:56.759000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Spam.msf, FALSE, 8fe6a10, TRUE)
> 2013-10-11 22:22:56.759000 UTC - 0[e0f140]: error opening db 80550006
It is perhaps "error is detected at (a). (b) and (c) is for rebuild-index.

In 14.pre.TB.log, following is seen.
At 21:45:28.923000, "3 hdrs in use" is logged for Spam.
> 2013-10-10 21:45:28.923000 UTC - 0[e0f140]: 85 open DB's
>(snip)
> 2013-10-10 21:45:28.923000 UTC - 0[e0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf - 3 hdrs in use
>(snip)
After it some MsgDaabases re closed, and Nagios.msf is openedagain  immediately.
This is peraps by "close notificaion to lisnter such as Virtual Folder, Filter" and "open request by listner".
> 2013-10-10 21:45:29.016000 UTC - 0[e0f140]: closing database    S:\data\ ... \edcint.co.nz\Bulk.sbd\Nagios.msf
> 2013-10-10 21:45:29.063000 UTC - 0[e0f140]: closing database    S:\data\ ... \edcint.co.nz\Bulk.sbd\MythTV.msf
> 2013-10-10 21:45:29.063000 UTC - 0[e0f140]: closing database    S:\data\ ... \edcint.co.nz\Bulk.sbd\GW.msf
> 2013-10-10 21:45:29.063000 UTC - 0[e0f140]: closing database    S:\data\ ... \edcint.co.nz\Bulk.sbd\CheckWMIPlus.msf
> 2013-10-10 21:45:29.063000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data ... \edcint.co.nz\Bulk.sbd\Nagios.msf, FALSE, ee2a10, TRUE)
At next "nn open DB's" log, "3 hdrs in use" is logged for Spam.
> 2013-10-10 21:45:29.079000 UTC - 0[e0f140]: 81 open DB's
>(snip)
> 2013-10-10 21:45:29.094000 UTC - 0[e0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf - 0 hdrs in use

Why  "NN hdrs in use" is reduced to NN=1 suddenly?
This may be a cause of "corrupted .msf after termination of Tb".

Comment 112

[WADA:World Anti-bad-Duping Agency]

Sent ==  S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Sent.msf ?

For edcint.co.nz\Sent:

In 14.Sent.Spam.TB.log, following is seen.
At 22:22:02.619000, \edcint.co.nz\Sent.msf is opened.
> 2013-10-11 22:22:01.009000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Sent.msf, FALSE, ee21c0, TRUE)
At 22:22:02.619000, in "NN open DB's" log, "242 hdrs in use" for \edcint.co.nz\Sent.msf
> 2013-10-11 22:22:02.619000 UTC - 0[e0f140]: 12 open DB's
>(snip)
> 2013-10-11 22:22:02.619000 UTC - 0[e0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Sent.msf - 242 hdrs in use
>(snip)
At 22:22:05.041000, in "NN open DB's" log, "241 hdrs in use" for \edcint.co.nz\Sent.msf
> 2013-10-11 22:22:05.041000 UTC - 0[e0f140]: 12 open DB's
>(snip)
> 2013-10-11 22:22:05.041000 UTC - 0[e0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Sent.msf - 241 hdrs in use
>(snip)
At 22:22:24.166000, in "NN open DB's" log, "1 hdrs in use" for \edcint.co.nz\Sent.msf
> 2013-10-11 22:22:24.166000 UTC - 0[e0f140]: 12 open DB's
>(snip)
> 2013-10-11 22:22:24.166000 UTC - 0[e0f140]: S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Sent.msf - 1 hdrs in use
> At  22:22:40.994000, close of \edcint.co.nz\Sent.msf is executed.
> 2013-10-11 22:22:40.994000 UTC - 0[e0f140]: closing database    S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Sent.msf
> At 22:22:57.072000,  open of \edcint.co.nz\Sent.msf is requested,
> 2013-10-11 22:22:57.072000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Sent.msf, FALSE, 727c410, TRUE)
> and \edcint.co.nz\Sent.msf is closed immediately.
> 2013-10-11 22:22:57.228000 UTC - 0[e0f140]: closing database    S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Sent.msf
> And \edcint.co.nz\Sent.msf is opened again,
> 2013-10-11 22:22:57.306000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Sent.msf, FALSE, 727c410, TRUE)
> thenlog for  \Temp\MozillaMailnews\Sent.msfis written(rebuil-index is invoked)
> 2013-10-11 22:22:57.384000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Sent.msf, FALSE, 727c670, TRUE)
> 2013-10-11 22:22:57.384000 UTC - 0[e0f140]: error opening db 80550006

Why  "NN hdrs in use" is reduced?
Why  "NN hdrs in use" is reduced to NN=1 suddenly?
Is sudden "1 hdrs in use" of \edcint.co.nz\Sent.msf relevant to internal rebuild-index upon Virtual Folder open?

By the way, have you tried to read log by yourself?
How can we know "when virual folders is clcked at folder pane by you" from log only?
How can we know "for which actual mail folder Tb tried to use C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Sent.msf" by log only?
Please write down "when did you do what", please. And please write "on which folder of which account rebuild-index was invoked", please.

Anyway, check Process Monitor log around "reduced HH hdrs in use" log, please.

Comment 113

[WADA:World Anti-bad-Duping Agency]

Anyway, you look to heve dicsovered consistent "steps to reproduce".
  (1) Restart Tb, open virtual folder just after restart, open virtual folder several times
      while Tb is running, and open virtual folder just before shutdown of Tb.
  (2) Repeat (1).
Even if no update of mail folder by "filter move, message purge etc.", and even when "outdated-msf condition" nor "no .msf file condition" doesn't exist, rebuild-index seems invoked upon Virtual Folder click, and it looks caused merely by repeated MsgDatabase open/close.

Comment 114

[WADA:World Anti-bad-Duping Agency]

Sorry, typo in comment #111.
"nn open DB's" log at 21:45:29.079000 for \Local Folders\Spam.msf is "0 hdrs in use".
("3 hdrs in use" was suddenly changed to "0 hdrs in use")

Comment 115

[WADA:World Anti-bad-Duping Agency]

Following is events seen in 14.Sent.Spam.TB.log and 14.Sent.Spam.PML on \Local Folders\Spam.msf and \Local Folders\Spam.

(1) Restart of Tb, start of log.
> 2013-10-11 22:21:02.650000 UTC - 0[e0f140]: mail.purge.min_delay=480 minutes
> 2013-10-11 22:21:02.650000 UTC - 0[e0f140]: mail.purge.timer_interval=5 minutes
> 2013-10-11 22:21:02.650000 UTC - 0[e0f140]: setting to check again in 5 minutes
> 2013-10-11 22:21:30.806000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Matthew\[e-mail]\Inbox.msf, FALSE, ee0b30, TRUE)
> 2013-10-11 22:21:30.869000 UTC - 0[e0f140]: 0 open DB's

(2-1) \Local Folders\Spam.msf is opened after restrt of Tb.
> 2013-10-11 22:21:54.947000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\ ... \Mail\Local Folders\Spam.msf, FALSE, 727e090, TRUE)
(2-2) Process Monitor log for Category=Write,Path contains \Spam(CSV format).
      While Tb is trying to open \Local Folders\Spam.msf, data is written to \Local Folders\Spam(perhaps by Junk move or Filter Move)
> "Time of Day"    ,"Path",                                 "Category","Operation"   ,"Result" ,"Detail"
> "7:21:55.1912890"."S:\Data\ ... \Local Folders\Spam",     "Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset: 4,429,449, Length: 5,862","2892","340","File System"
> "7:21:55.1940361"."S:\Data\ ... \Local Folders\Spam",     "Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset: 4,427,776, Length: 8,192, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O","2892","340","File System"
(2-3) Process Monitor log for Category=Write,Path contains \Spam(CSV format).
      \Local Folders\Spam.msf is updated.
> "Time of Day"    ,"Path",                                 "Category","Operation"   ,"Result" ,"Detail"
> "7:21:55.2098372"."S:\Data\ ... \Local Folders\Spam.msf", "Write"   ,"FASTIO_WRITE","SUCCESS","Offset:   119,514, Length:   960","2892","340","File System"
> "7:21:55.2564840"."S:\Data\ ... \Local Folders\Spam.msf", "Write"   ,"FASTIO_WRITE","SUCCESS","Offset:   120,474, Length:   310","2892","340","File System"
> "7:21:55.2566703"."S:\Data\ ... \Local Folders\Spam.msf", "Write"   ,"FASTIO_WRITE","SUCCESS","Offset:   120,784, Length:    38","2892","340","File System"
> "7:21:55.2573998"."S:\Data\ ... \Local Folders\Spam.msf", "Write"   ,"FASTIO_WRITE","SUCCESS","Offset:   120,822, Length:    22","2892","340","File System"
> "7:21:55.6267293"."S:\Data\ ... \Local Folders\Spam.msf", "Write"   ,"FASTIO_WRITE","SUCCESS","Offset:   120,844, Length:    22","2892","340","File System"
> "7:21:56.2848831"."S:\Data\ ... \Local Folders\Spam.msf", "Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset:   118,784, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O","2892","340","File System"
(2-4) After it, \Local Folders\Spam.msf is closed. 
> 2013-10-11 22:21:56.306000 UTC - 0[e0f140]: closing database    S:\data\ ... \Mail\Local Folders\Spam.msf

(3) "Open \Local Folders\Spam.msf, then close \Local Folders\Spam.msf immediately" is repeated.
> 2013-10-11 22:22:02.588000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\ ... \Mail\Local Folders\Spam.msf, FALSE, 4fb2550, TRUE)
> 2013-10-11 22:22:02.666000 UTC - 0[e0f140]: closing database    S:\data\ ... \Mail\Local Folders\Spam.msf
> 2013-10-11 22:22:05.009000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\ ... \Mail\Local Folders\Spam.msf, FALSE, ee2090, TRUE)
> 2013-10-11 22:22:05.072000 UTC - 0[e0f140]: closing database    S:\data\ ... \Mail\Local Folders\Spam.msf
> 2013-10-11 22:22:46.463000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\ ... \Mail\Local Folders\Spam.msf, FALSE, 727c410, TRUE)
> 2013-10-11 22:22:46.588000 UTC - 0[e0f140]: closing database    S:\data\ ... \Mail\Local Folders\Spam.msf
> 2013-10-11 22:22:46.603000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\ ... \Mail\Local Folders\Spam.msf, FALSE, 727c410, TRUE)
> 2013-10-11 22:22:46.697000 UTC - 0[e0f140]: closing database    S:\data\ ... \Mail\Local Folders\Spam.msf

(4-1) Open \Local Folder\s\Spam.msf and close \Local Folders\Spam.msf immediately,
    then re-open \Local Folder\s\Spam.msf,
    and "error opening db 80550006" is logged for1\Temp\MozillaMailnews\Spam.msf.
    It's perhaps "Virtual Folder is clicked and rebuild-index was invoked".
> 2013-10-11 22:22:56.463000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\ ... \Mail\Local Folders\Spam.msf, FALSE, 8fe68e0, TRUE)
> 2013-10-11 22:22:56.572000 UTC - 0[e0f140]: closing database    S:\data\ ... \Mail\Local Folders\Spam.msf
> 2013-10-11 22:22:56.681000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\ ... \Mail\Local Folders\Spam.msf, FALSE, 8fe68e0, TRUE)
> 2013-10-11 22:22:56.759000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Spam.msf, FALSE, 8fe6a10, TRUE)
> 2013-10-11 22:22:56.759000 UTC - 0[e0f140]: error opening db 80550006
(4-2) Process Monitor log for Category=Write,Path contains \Spam(CSV format).
      \Local Folders\Spam.msf is updated.
 (note: it's not "re-write of entire Spam.msf data". it's "update of required part" only.) 
> "Time of Day"    ,"Path",                                 "Category","Operation"   ,"Result" ,"Detail"
> "7:22:56.9290643"."S:\Data\ ... \Local Folders\Spam.msf", "Write"   ,"FASTIO_WRITE","SUCCESS","Offset:   120,866, Length: 3,223","2892","340","File System"
> "7:22:56.9445658"."S:\Data\ ... \Local Folders\Spam.msf", "Write"   ,"FASTIO_WRITE","SUCCESS","Offset:   124,089, Length:    22","2892","340","File System"
> "7:22:57.0088678"."S:\Data\ ... \Local Folders\Spam.msf", "Write"   ,"FASTIO_WRITE","SUCCESS","Offset:   124,111, Length:   124","2892","340","File System"
> "7:23:03.8018289"."S:\Data\ ... \Local Folders\Spam.msf", "Write"   ,"FASTIO_WRITE","SUCCESS","Offset:   124,235, Length:    22","2892","340","File System"
> "7:23:03.8024347"."S:\Data\ ... \Local Folders\Spam.msf", "Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset:   122,880, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O","2892","340","File System"
> "7:26:03.1837176"."S:\Data\ ... \Local Folders\Spam.msf", "Write"   ,"FASTIO_WRITE","SUCCESS","Offset:   124,257, Length:    74","2892","340","File System"
> "7:26:03.1839007"."S:\Data\ ... \Local Folders\Spam.msf", "Write"   ,"FASTIO_WRITE","SUCCESS","Offset:   124,331, Length:    22","2892","340","File System"
> "7:26:03.1847265"."S:\Data\ ... \Local Folders\Spam.msf", "Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset:   122,880, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O","2892","340","File System"
(4-3) Tnen l\Local Folders\Spam.msf is closed.
> 2013-10-11 22:23:03.838000 UTC - 0[e0f140]: closing database    S:\data\ ... \Mail\Local Folders\Spam.msf
(4-4) Following is Process Moitor log of Category=Write for \Local Folders\Spam.sbd\SpamAssassin SPAM.msf.
      Last "72 bytes + 22 bytes" is perhaps "timestamp and file size" for outdated-msf condition detection.
> "Time of Day"    ,"Path",                                                      "Category","Operation"   ,"Result" ,"Detail"
> "7:26:03.3337533"."S:\Data\ ... \Local Folders\Spam.sbd\SpamAssassin SPAM.msf","Write"   ,"FASTIO_WRITE","SUCCESS","Offset:     2,742, Length:    74","2892","340","File System"
> "7:26:03.3339115"."S:\Data\ ... \Local Folders\Spam.sbd\SpamAssassin SPAM.msf","Write"   ,"FASTIO_WRITE","SUCCESS","Offset:     2,816, Length:    22","2892","340","File System"
> "7:26:03.3345261"."S:\Data\ ... \Local Folders\Spam.sbd\SpamAssassin SPAM.msf","Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset:         0, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O","2892","340","File System"

Problem like following?
  While Tb is trying to open \Local Folders\Spam.msf just after restart of Tb,
  "Filer move or Junk move" happens on \Local Folders\Spam.msf
  upon first new mail check and mail download after restart of Tb.
  Because MsgDatabase is not opened yet,
  "Filer move or Junk move" appends mail data to \Local Folders\Spam
  without updating MsgDatabase content correctly.
  Then inconcistency in MsgDatabese content(.msf file content) is produced.

Comment 116

[WADA:World Anti-bad-Duping Agency]

To Matthew Jurgens8bug opener)]
What data is writte at "Offset: 4,429,449, Length: 5,862" of file named \Local Folders\Spam?
Because of "local mail folder file", timestamp of "From -" line is download or copy time of mail(local time without timezone). If corresponding data in \Local Folders\Spam is "data of a mail", what timestamp is writen in From -" line?

Comment 117

[WADA:World Anti-bad-Duping Agency]

Following is events seen in 14.Sent.Spam.TB.log and 14.Sent.Spam.PML on \edcint.co.nz\Sent.msf and \edcint.co.nz\Sent.
Log is pretty similat to \Local Folders\Spam.msf/\Local Folders\Spam case, although it's slightly different.

(1) Restart of Tb, start of log.
> 2013-10-11 22:21:02.650000 UTC - 0[e0f140]: mail.purge.min_delay=480 minutes
> 2013-10-11 22:21:02.650000 UTC - 0[e0f140]: mail.purge.timer_interval=5 minutes
> 2013-10-11 22:21:02.650000 UTC - 0[e0f140]: setting to check again in 5 minutes
> 2013-10-11 22:21:30.806000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Matthew\[e-mail]\Inbox.msf, FALSE, ee0b30, TRUE)
> 2013-10-11 22:21:30.869000 UTC - 0[e0f140]: 0 open DB's

(2-1)  \edcint.co.nz\Sent.msf is opened after resatrt of Tb.
> 2013-10-11 22:22:01.009000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\ ... \edcint.co.nz\Sent.msf, FALSE, ee21c0, TRUE)
(2-2) Process Monitor log for Category=Write,Path contains \Sent(CSV format).
      While Tb is trying to open \edcint.co.nz\Sent.msf,
      data is written to \edcint.co.nz\Sent.msf.
      (perhaps by Junk move or Filter Move or Send mail)
>            "Time of Day"    ,"Path"                               ,"Category","Operation"   ,"Result" ,"Detail"
>            "7:22:01.3310949","S:\Data\ ... \edcint.co.nz\Sent"    ,"Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset: 95,116,578, Length:  8,014"
>            "7:22:01.3381851","S:\Data\ ... \edcint.co.nz\Sent"    ,"Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset: 95,113,216, Length: 12,288, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O"
(2-3) Process Monitor log for Category=Write,Path contains \Sent(CSV format).
      \edcint.co.nz\Sent.msf is updated.
>            "Time of Day"    ,"Path"                               ,"Category","Operation"   ,"Result" ,"Detail"
>            "7:22:01.3546760","S:\Data\ ... \edcint.co.nz\Sent.msf","Write"   ,"FASTIO_WRITE","SUCCESS","Offset:    454,475, Length:    938"
>            "7:22:01.4085874","S:\Data\ ... \edcint.co.nz\Sent.msf","Write"   ,"FASTIO_WRITE","SUCCESS","Offset:    455,413, Length:    291"
>            "7:22:01.4088091","S:\Data\ ... \edcint.co.nz\Sent.msf","Write"   ,"FASTIO_WRITE","SUCCESS","Offset:    455,704, Length:     41"
>            "7:22:01.4096870","S:\Data\ ... \edcint.co.nz\Sent.msf","Write"   ,"FASTIO_WRITE","SUCCESS","Offset:    455,745, Length:     24"
(2-4) While updating \edcint.co.nz\Sent.msf,
      "242 hdrs in use" is loged for \edcint.co.nz\Sent.msf.
> 2013-10-11 22:22:02.619000 UTC - 0[e0f140]: 12 open DB's
> 2013-10-11 22:22:02.619000 UTC - 0[e0f140]: S:\data\ ... \edcint.co.nz\Sent.msf - 242 hdrs in use
(2-4) While updating \edcint.co.nz\Sent.msf,
      "241 hdrs in use" is loged for \edcint.co.nz\Sent.msf.
> 2013-10-11 22:22:05.041000 UTC - 0[e0f140]: 12 open DB's
> 2013-10-11 22:22:05.041000 UTC - 0[e0f140]: S:\data\ ... \edcint.co.nz\Sent.msf - 241 hdrs in use
(2-5) While updating \edcint.co.nz\Sent.msf,
      "1 hdrs in use" is loged for \edcint.co.nz\Sent.msf.
> 2013-10-11 22:22:24.166000 UTC - 0[e0f140]: 12 open DB's
> 2013-10-11 22:22:24.166000 UTC - 0[e0f140]: S:\data\ ... \edcint.co.nz\Sent.msf - 1 hdrs in use
(2-6) Process Monitor log for Category=Write,Path contains \Sent(CSV format).
      \edcint.co.nz\Sent.msf is still updated.
>            "Time of Day"    ,"Path"                               ,"Category","Operation"   ,"Result" ,"Detail"
>            "7:22:26.1554226","S:\Data\ ... \edcint.co.nz\Sent.msf","Write"   ,"FASTIO_WRITE","SUCCESS","Offset:    455,793, Length:     24"
>            "7:22:26.1560142","S:\Data\ ... \edcint.co.nz\Sent.msf","Write"   ,"FASTIO_WRITE","SUCCESS","Offset:    455,817, Length:     24"
>            "7:22:34.6486371","S:\Data\ ... \edcint.co.nz\Sent.msf","Write"   ,"FASTIO_WRITE","SUCCESS","Offset:    455,841, Length:     24"
>            "7:22:34.6490061","S:\Data\ ... \edcint.co.nz\Sent.msf","Write"   ,"FASTIO_WRITE","SUCCESS","Offset:    455,865, Length:     24"
>            "7:22:36.0281345","S:\Data\ ... \edcint.co.nz\Sent.msf","Write"   ,"FASTIO_WRITE","SUCCESS","Offset:    455,889, Length:     63"
>            "7:22:40.9514623","S:\Data\ ... \edcint.co.nz\Sent.msf","Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset:    454,656, Length:  4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O"
(2-7) After it, \edcint.co.nz\Sent.msf is closed.
> 2013-10-11 22:22:40.994000 UTC - 0[e0f140]: closing database    S:\data\ ... \edcint.co.nz\Sent.msf

(3) Open \edcint.co.nz\Sent.msf, then close \edcint.co.nz\Sent.msf immediately.
> 2013-10-11 22:22:48.791000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\ ... \edcint.co.nz\Sent.msf, FALSE, 727e550, TRUE)
> 2013-10-11 22:22:48.916000 UTC - 0[e0f140]: closing database    S:\data\ ... \edcint.co.nz\Sent.msf

(4-1) Open \edcint.co.nz\Sent.msf and close \edcint.co.nz\Sent.msf immediately,
    then re-open \edcint.co.nz\Sent.msf,
    and "error opening db 80550006" is logged for1\Temp\MozillaMailnews\Sent.msf.
    It's perhaps "Virtual Folder is clicked and rebuild-index was invoked".
> 2013-10-11 22:22:57.072000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\ ... \edcint.co.nz\Sent.msf, FALSE, 727c410, TRUE)
> 2013-10-11 22:22:57.228000 UTC - 0[e0f140]: closing database    S:\data\ ... \edcint.co.nz\Sent.msf
> 2013-10-11 22:22:57.306000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\data\ ... \edcint.co.nz\Sent.msf, FALSE, 727c410, TRUE)
> 2013-10-11 22:22:57.384000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Sent.msf, FALSE, 727c670, TRUE)
> 2013-10-11 22:22:57.384000 UTC - 0[e0f140]: error opening db 80550006
(4-2) Process Monitor log for Category=Write,Path contains \Spam(CSV format).
      \edcint.co.nz\Sent.msf is updated.
      (note: it's not "re-write of entire Spam.msf data".) 
      (      it's "update of required part" only.        )
>            "7:23:00.8316523","S:\Data\ ... \edcint.co.nz\Sent.msf","Write"   ,"FASTIO_WRITE","SUCCESS","Offset:    455,952, Length:  2,599"
>            "7:23:00.8365133","S:\Data\ ... \edcint.co.nz\Sent.msf","Write"   ,"FASTIO_WRITE","SUCCESS","Offset:    458,551, Length:     24"
>            "7:23:00.8829936","S:\Data\ ... \edcint.co.nz\Sent.msf","Write"   ,"FASTIO_WRITE","SUCCESS","Offset:    458,575, Length:    104"
>            "7:23:03.8042540","S:\Data\ ... \edcint.co.nz\Sent.msf","Write"   ,"FASTIO_WRITE","SUCCESS","Offset:    458,679, Length:     24"
>            "7:23:03.8048615","S:\Data\ ... \edcint.co.nz\Sent.msf","Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset:    454,656, Length:  4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O"
(4-3) Tnen \edcint.co.nz\Sent.msf is closed.
> 2013-10-11 22:23:03.838000 UTC - 0[e0f140]: closing database    S:\data\ ... \edcint.co.nz\Sent.msf

(5) After this, no log relevant to \edcint.co.nz\Sent.msf is seen.

To Matthew Jurgens(bug opener):
What data is writte at "Offset: 95,116,578, Length:  8,014" of file named \edcint.co.nz\Sent?

Comment 118

[Matthew Jurgens]

(In reply to WADA from comment #111)
> Spam == S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local
Yes

(In reply to WADA from comment #112)
> Sent ==  S:\data\Mozilla
> Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Sent.msf ?

(In reply to WADA from comment #112)
> How can we know "for which actual mail folder Tb tried to use
> C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Sent.msf" by log only?
This will be hard for folders like Sent that are in multiple accounts. Folders like Spam will be easy since there is only one of those

> Please write down "when did you do what", please. 
Yes

>And please write "on which
> folder of which account rebuild-index was invoked", please.
This will be hard for folders like Sent that are in multiple accounts. Folders like Spam will be easy since there is only one of those


> Anyway, check Process Monitor log around "reduced HH hdrs in use" log,
> please.

(In reply to WADA from comment #113)
> Anyway, you look to heve dicsovered consistent "steps to reproduce".
It is not consistent. I can not guarantee a rebuild not can I predict which folder will rebuild.


(In reply to WADA from comment #116)
> What data is writte at "Offset: 4,429,449, Length: 5,862" of file named
> \Local Folders\Spam?
This is the last mail that has been moved to the Spam folder. It starts with the From line:
From - Sat Oct 12 09:21:54 2013

(In reply to WADA from comment #117)
> What data is writte at "Offset: 95,116,578, Length:  8,014" of file named
> \edcint.co.nz\Sent?
This looks like it is the last mail that would have been moved into the Sent folder. 
This particular mail is actually moved to the Sent folder from the Inbox by a filter rule that I have. So it is not the traditional sent mail that is moved by TB after sending a mail.
It starts as follows:
From - Sat Oct 12 09:22:00 2013
X-Account-Key: account4
X-UIDL: UID70624-1159685244
X-Mozilla-Status: 0011
X-Mozilla-Status2: 00000000
X-Mozilla-Keys: $label1                                                                         
Return-path: <[e-mail]>
Envelope-to: [e-mail]
Delivery-date: Thu, 10 Oct 2013 18:05:17 -0500
Received: from pa49-183-3-208.pa.vic.optusnet.com.au ([49.183.3.208]:41818 helo=[10.128.244.40])
	by tilde.accountservergroup.com with esmtpa (Exim 4.80)
	(envelope-from <[e-mail]>)
	id 1VUPIB-0000e5-7Y; Thu, 10 Oct 2013 18:05:17 -0500
Date: Fri, 11 Oct 2013 10:04:37 +1100

Comment 119

[WADA:World Anti-bad-Duping Agency]

Problem looks;
1. FolderX.msf is not opened yet, or is already closed.
2. New mail is downloaded, and is moved to FolderX by Message Filter.
3. While Tb is reading entire data in FolderX.msf and constructing MsgDatabase for FolderX,
   message filter obtains MsgDBHdr key for the moved mail in FolderX,
   and message filter appends mail data to file named FolderX.
   Then message filter updates MsgDBHdr data(meta data of mail) of the mail in FolderX.
   Note: This is not "message filter only behavior".
         This is applicable to Mail Copy/Move(via menu. not Drag&Drop), Sent mail copy,
         Draft/Template save, Archive.
         This is perhaps applicable to "Junk move", "message purge" etc.
4. Because "file open of FolderX.msf" is already completed, updated MsgDatabase part
   is physically written to FolderX.msf file.
5. FolderX.msf is closed.
6. If FolderX.msf is big and is located on file server, it takes pretty long to read
   entire FolderX.msf file content.
   Then FolderX.msf is corrupted during 3/4/5.
7. Because FolderX.msf is corrupted, FolderX.msf is opened and closed immediately
   upon each simple access to MsgDatabase by some internal activities.
8. When mail folder named FolderX is accessed via Virtual Folder,
   or when mail folder named FolderX is explicitly opened by "folder click at folder pane",
   Rebuild-Index is invoked.

"Steps to reproduce" based on above assumption.
(0) Create VirtualFolderX. Search Target=FolderX of an POP3 account.
    Create message filter which moves mail to FolderX.
    FolderX/FolderX.msf is located on file server.
(1) Restart Tb, open VirtualFolderX, confirm "no rebuild-index",
    click Inbox in order to force "last selected folder != FolderX", and terminate Tb.
(2) Send mail which will be moved to FolderX.
(3) Restart Tb with logging enabled, with Process Monitor enabled.
(4) After new mail is download and is moved to FolderX,
    click VirtualFolderX at folder pane => Rebuld-Index happens.

Please check followings.
(0) VirtualFolderX : Search Target = FolderA and FolderB
    (please use unique name for ease of problem determination and log reading)

(1) Get log with very small mail folder.
    Copy some small mails to FolderA. File size of FolderA.msf == around 20 KB.
    Because Tb reads .msf file with 8KB buffer, Process Monitor log for the read is small.
    It's convenient to read Process Monitor log to know Tb's behavior.
    Terminate Tb, send 3 small mails which will be moved to FolderA,
    restart Tb and keep backup of FolderA.msf after mail download,
    and click VirtualFolderX.

    What data is written in FolderA and backup of FolderA.msf at "Offset: aa, Length: bb"
    of Process Monitor log of "Category begins with Write" for FolderA and FolderA.msf? 

(2) Get log with big mail folder.
    Copy all mails in \edcint.co.nz\Sent to FolderB.
    Terminate Tb, send 3 small mails which will be moved to FolderB,
    restart Tb and keep backup of FolderB.msf after mail download,
    and click VirtualFolderX.

    Is rebuild-index invoked by open of VirtualFolderX?
    What data is written in FolderB and backup of FolderB.msf at "Offset: aa, Length: bb"
    of Process Monitor log of "Category begins with Write" for FolderB and FolderB.msf?

FYI.
After filterling Process Monitor log with appropriate columns, save log data as ".CSV" file or ".XML" file. You can read and edit log data by Text Editor.

Comment 120

[Matthew Jurgens]

(In reply to WADA from comment #119)
> "Steps to reproduce" based on above assumption.
So far, I have been unable to reproduce a single folder index rebuild by trying these steps.
The folder is only small. So I may try making it larger.

Comment 121

[Matthew Jurgens]

(In reply to Matthew Jurgens from comment #120)
> The folder is only small. So I may try making it larger.
Even for a large folder, I have been unable to reproduce a single folder index rebuild by trying these steps (using the test folders)

Comment 122

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #121)
> Even for a large folder, I have been unable to reproduce a single folder
> index rebuild by trying these steps (using the test folders)
It sounds that other conditions is needed, for example, download from multiple servers at same time and filter move to big mboxes in multiple accounts at same time.

Anyway, keep opening Virtual Folder several times in daily use, please.
There is no need of NSPR log nor Process Monitor log, until 14.Sent.Spam.PML will be sufficiently analyzed.
If rebuild-index happened, see message filter log(filterlog.html), and check "how many messages are moved by filter to what Mboxes at when since restart of Tb", please.

(In reply to Matthew Jurgens from comment #120)
> > "Steps to reproduce" based on above assumption.
> So far, I have been unable to reproduce a single folder index rebuild by
> trying these steps. The folder is only small.
Can you show us NSPR log and Process Monitor log for it?
Because of "download of a few mails only at single account only" and "open/read/write/close of two small file only upon filter move", log size is small. So, it's helpful to understand Tb's normal behavior upon filter move.

By the way, do you enable Quarantine option?
- Tools/Options/Security/Anti-Virus
  mailnews.downloadToTempFile = true(Checked/Enabled) / false(Unchecked/Disabled)
If false, mail data is downloaded to Inbox, and mail is copied to move target folder by filter move, then Inbox is truncated to original file size in order to remove moved mail.
If true, mail data is downloaded to temp file, and mail is copied to move target folder by filter move, then temp file is deleted.
So, it's need to check different file access in Process Monitor log.

Comment 123

[Matthew Jurgens]

(In reply to WADA from comment #122)
> Can you show us NSPR log and Process Monitor log for it?
http://www.edcint.co.nz/LargeFiles/Completed/16.SmallTest.Filtered.PML.zip
http://www.edcint.co.nz/LargeFiles/Completed/16.SmallTest.TB.log

> By the way, do you enable Quarantine option?
> - Tools/Options/Security/Anti-Virus
This is set to false

Comment 124

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #123)
> 16.SmallTest.Filtered.PML.zip
> 16.SmallTest.TB.log

Thanks for log, but...
Several mails look to have been moved to TestFolderA, but TestFolderA is huge folder...
TestFolderB is pretty small folder(size of TestFolderB.msf=2469), but nothing is written to TestFolderB...

Do you understand reason why I asked you "small log"?
Have you tried to read all Process Monitor logs filtered with "Path contains \TestFolderA" or "Path contains \Spam" by yourself in order to know "what Tb does do when Tb moves mail by filter" or "what Tb did on Spam.msf and Spam upon filter move before Tb invoked rebuild-index of Spam folder"?

Because "write log to Inbox", "read log of Inbox followed by write log of move target folder and truncate log of Inbox" etc. are held in Process Monitor log, "when, from which Inbox, to which folder, was filter move done by Tb" can be known from Process Monitor log. However, "reading many many log lines with trial and error on file name etc." is needed for it.
Please write at least "you did download of how many mails to Inbox under which directory or server/account at when" and "mails are moved to which folder" and "you opened Virtual Folder or move target folder at when".
Please note that "to know what happened from log for open/read/write of file only" is hard, but "to know this open/read/write log was produced by what kind of activity from your explanation with timestamp" is easy.

Comment 125

[WADA:World Anti-bad-Duping Agency]

Attachment: Log for \Spam in 14.Sent.Spam.PML

Following is relevant to problem?
  "EndOfFile: 1601/01/01 9:00:00" upon close of \Spam after appending data
If so, why "error opening db" for Outdated-msf condition is not logged by Tb's MSDB:5?
Phenomenon is following?
  ZERO is returned upon \Spam close, and Size=ZERO is written in \Spam.msf.
  When subsequent open of \Spam, EndOfFile=0 is returned to Query for \Spam
  so Outdated-msf conditin doesn't occur.
  After a while, correct EndOfFile is returned to Query for \Spam.
  As "Virtual Folder open" correctly handle mismatch between Size in \Spam.msf
  and returned correct EndOfFile, or "Virtual Folder open" does do additional
  checks when Size=0 in \Spam.msf, Rebuild-index is correctly invoked.

Comment 126

[WADA:World Anti-bad-Duping Agency]

Attachment: Log for \Spam in 14.Sent.Spam.PML(path is shortend for ease of reading)

Comment 127

[kev]

(In reply to WADA from comment #122)
> It sounds that other conditions is needed, for example, download from
> multiple servers at same time and filter move to big mboxes in multiple
> accounts at same time.

If you mean this in general, I want to point out that these conditions are not present in my case: single server, no filters.

Comment 128

[WADA:World Anti-bad-Duping Agency]

Attachment: Log for QUERY_INFORMATION of \Spam in 14.Sent.Spam.PML(path is shortend for ease of reading) (

"EndOfFile: 4,429,449" is returned to following Type: QueryStandardInformationFile only.
  7:21:55.1908643
  Type: QueryStandardInformationFile,
  AllocationSize: 5,242,880,
  EndOfFile: 4,429,449,
  NumberOfLinks: 1,
  DeletePending: False,
  Directory: Fals
All other log for Query is following Type:, and "EndOfFile: 1601/01/01 9:00:00" is always returned.
  Type: QueryNetworkOpenInformationFile
  EndOfFile: 1601/01/01 9:00:00
This is SPEC of SMB1?

Comment 129

[WADA:World Anti-bad-Duping Agency]

QueryStandardInformationFile is perhaps used for;
  "Filter move" requests write of Length: 5,862 at EndOfFile(append operation) of \Spam.
  Because "write at EndOfFile", OS requests QueryStandardInformationFile, and OS gets
  actual EndOfFile/AllocationSize which is perhaps obtained from server if SMB1.
  Then, OS physically writes data to HDD(if SMB1, this is converted to request to server).
    Read data from EndOfFile: 4,429,449 to AllocationSize: 5,242,880
    Replace data at Offset: 4,429,449, Length: 5,862 by requested data by Tb.
    Write data of Offset: 4,427,776, Length: 8,192 (== Sector Size * N if HDD)

Because all other Query'es which returns EndOfFile, including Query issued just before update of \Spam.msf which is executed after above append operation to \Spam, is QueryNetworkOpenInformationFile, and File Size passed to Tb by Query operation is always "EndOfFile: 1601/01/01 9:00:00". The "1601/01/01 9:00:00==ZERO" is perhaps always written as "File Size of \Spam" in \Spam.msf.
If so, because "File Size in \Spam.msf(ZERO)" == "EndOfFile: 1601/01/01 9:00:00(ZERO)" always, Outdated-msf condition won't occur.

If "File Size in \Spam.msf(ZERO)" is simply reason of rebuild-index, rebuild-index should occur upon any "folder open by \Spam folder click" or "folder access to \Spam by Virtual Folder" after filter move to \Spam happened.

Why rebuild-index occurred on \Spam and \Sent when 14.Sent.Spam.TB.log/14.Sent.Spam.PML was gotten, even though rebuild-index usually doesn't occur? 
If \Spam folder is explicitly opened(\Spam is clicked at folder pane) after filter move to \Spam, problem won't occur?
"Virtual Folder open just after filter move to \Spam" only problem?
If so, why rebuild-index didn't occur on TestFolderA when you got 16.SmallTest.Filtered.PML.zip and 16.SmallTest.TB.log?

Comment 130

[WADA:World Anti-bad-Duping Agency]

No "error opening db" may be difference among modules which is used for MsgDatabase open request.

(A) nsMsgDatabase::OpenInternal -> nsMsgDatabase::OpenMDB
> http://mxr.mozilla.org/comm-central/search?string=error+opening+db&find=&findi=&filter=^[^\0]*%24&hitlimit=&tree=comm-central
> "error opening db" is written by nsMsgDatabase.cpp#1232 only.
>   http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#1232
>   OpenMDB() is used just before it(nsMsgDatabase.cpp#1230).
> OpenMDB() is used in nsMsgDatabase::OpenInternal
>   http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#1219
> 
> OpenMDB() is defined at :
>   http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#1331
> OpenMDB() is used at :
>   http://mxr.mozilla.org/comm-central/ident?i=OpenMDB

(B) nsMsgLocalMailFolder::GetDatabaseWOReparse
> http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsLocalMailFolder.cpp#274
>  273 //we treat failure as null db returned
>  274 NS_IMETHODIMP nsMsgLocalMailFolder::GetDatabaseWOReparse(nsIMsgDatabase **aDatabase)
> GetDatabaseWOReparse() is called at :
>  http://mxr.mozilla.org/comm-central/ident?i=GetDatabaseWOReparse
> A GetDatabaseWOReparse() call example.
>  http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsLocalMailFolder.cpp#203
>  203 // this won't force a reparse of the folder if the db is invalid.
>  204 NS_IMETHODIMP
>  205 nsMsgLocalMailFolder::GetMsgDatabase(nsIMsgDatabase** aMsgDatabase)
>  206 {
>  207   return GetDatabaseWOReparse(aMsgDatabase);
>  208 }

(C) nsMsgLocalMailFolder::GetDatabaseWithReparse
> http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsLocalMailFolder.cpp#300
>  297 // Makes sure the database is open and exists.  If the database is out of date,
>  298 // then this call will run an async url to reparse the folder. The passed in
>  299 // url listener will get called when the url is done.
>  300 NS_IMETHODIMP nsMsgLocalMailFolder::GetDatabaseWithReparse(nsIUrlListener *aReparseUrlListener, nsIMsgWindow *aMsgWindow,
> GetDatabaseWithReparse is called at :
>   http://mxr.mozilla.org/comm-central/ident?i=GetDatabaseWithReparse
> http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsLocalMailFolder.cpp#1850
> GetDatabaseWithReparse call examples.
> Upon GetNewMessages.
> http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsLocalMailFolder.cpp#1850
>  1850 NS_IMETHODIMP nsMsgLocalMailFolder::GetNewMessages(nsIMsgWindow *aWindow, nsIUrlListener *aListener)
>  1878     rv = localInbox->GetDatabaseWithReparse(nullptr, aWindow, getter_AddRefs(db));
> In SendLater processing, perhaps when special condition.
> http://mxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgSendLater.cpp#767
>  766 // Returns NS_OK if the db is OK, an error otherwise, e.g., we had to reparse.
>  767 nsresult nsMsgSendLater::ReparseDBIfNeeded(nsIUrlListener *aListener)
>  775   return locFolder->GetDatabaseWithReparse(aListener, nullptr,

If other than (A) is used, "error opening db" is not logged, even when other conditions such as NS_ERROR_NOT_INITIALIZED is returned.
Vrtual Folder relevant routine looks to check NS_ERROR_NOT_INITIALIZED condition.
> http://mxr.mozilla.org/comm-central/source/mailnews/base/src/nsMsgXFVirtualFolderDBView.cpp#292
>  292 nsMsgXFVirtualFolderDBView::OnSearchDone(nsresult status)
>  293 {
>  294   NS_ENSURE_TRUE(m_viewFolder, NS_ERROR_NOT_INITIALIZED);
FileSize=ZERO may produce NS_ERROR_NOT_INITIALIZED condition in "Search by Virtual Folder".

Comment 131

[WADA:World Anti-bad-Duping Agency]

In Virtul Folder, (B) GetMsgDatabase() was used for each Search Target folder.
> http://mxr.mozilla.org/comm-central/source/mailnews/base/src/nsMsgXFVirtualFolderDBView.cpp#351
>  351 nsMsgXFVirtualFolderDBView::OnNewSearch()
>  406 for (int32_t i = 0; i < scopeCount; i++)
>  417       nsresult rv = searchFolder->GetMsgDatabase(getter_AddRefs(searchDB))

Comment 132

[WADA:World Anti-bad-Duping Agency]

If "EndOfFile: 1601/01/01 9:00:00" is saved in \Spam.msf and is identical to ZERO for Tb, and If nsMsgDatabase::CheckForErrors is called, summaryFileExists = false is set.
summaryFileExists==false may invoke rebuild-index.
> http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#1251
> 1251 nsresult nsMsgDatabase::CheckForErrors(nsresult err, bool sync,
> 1252                                        nsIFile *summaryFile)
> 1253 {
> 1254   nsCOMPtr<nsIDBFolderInfo> folderInfo;
> 1255   bool summaryFileExists;
> 1256   bool newFile = false;
> 1257   bool deleteInvalidDB = false;
> 1258 
> 1259   bool exists;
> 1260   int64_t fileSize;
> 1261   summaryFile->Exists(&exists);
> 1262   summaryFile->GetFileSize(&fileSize);
> 1263   // if the old summary doesn't exist, we're creating a new one.
> 1264   if ((!exists || !fileSize) && m_create)
> 1265     newFile = true;
> 1266 
> 1267   summaryFileExists = exists && fileSize > 0;

Comment 133

[WADA:World Anti-bad-Duping Agency]

CheckForErrors() was called in (A) nsMsgDatabase::OpenInternal.
> http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#1219
> 1219 nsresult nsMsgDatabase::OpenInternal(nsIFile *summaryFile, bool aCreate,
> 1230   nsresult rv = OpenMDB(summaryFilePath.get(), aCreate, sync);
> 1231   if (NS_FAILED(rv))
> 1232     PR_LOG(DBLog, PR_LOG_ALWAYS, ("error opening db %lx", rv));
> 1248   return CheckForErrors(rv, true, summaryFile);

If summaryFileExists==false is set in CheckForErrors(), NS_MSG_ERROR_FOLDER_SUMMARY_MISSING is returned from CheckForErrors(), then the error is returned to caller of (A) nsMsgDatabase::OpenInternal.
> http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#1251
> 1251 nsresult nsMsgDatabase::CheckForErrors(nsresult err, bool sync,
> 1267   summaryFileExists = exists && fileSize > 0; 
> 1323   return (summaryFileExists) ? err : NS_MSG_ERROR_FOLDER_SUMMARY_MISSING;
Because CheckForErrors() is called after "OpenMDB() call with 'error opening db' log if error" in (A) nsMsgDatabase::OpenInternal, log of "error opening db 80550005" is not written even when NS_MSG_ERROR_FOLDER_SUMMARY_MISSING condition is raised by CheckForErrors(), even when (A) nsMsgDatabase::OpenInternal is used for MsgDatabase or MsgFolder open.

CheckForErrors() is called in nsMsgDBService::OpenMore too.
> http://mxr.mozilla.org/comm-central/search?string=CheckForErrors&find=mail
> http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#222
>  222 NS_IMETHODIMP nsMsgDBService::OpenMore(nsIMsgDatabase *aDB,
>  264         rv = msgDatabase->CheckForErrors(rv, false, summaryFile);
However, no one looks to call OpenMore().
> http://mxr.mozilla.org/comm-central/ident?i=OpenMore

Comment 134

[WADA:World Anti-bad-Duping Agency]

If followins is cause,
- "EndOfFile: 1601/01/01 9:00:00" is saved in \Spam.msf and is identical to ZERO for Tb.
- nsMsgDatabase::CheckForErrors is called.
- nsMsgDatabase::CheckForErrors sets summaryFileExists=false
  because FileSize in \Spam.msf=="EndOfFile: 1601/01/01 9:00:00"==ZERO,
  then NS_MSG_ERROR_FOLDER_SUMMARY_MISSING is raised.
When, How, by Whom, is nsMsgDatabase::CheckForErrors called then NS_MSG_ERROR_FOLDER_SUMMARY_MISSING is raised?

If NS_MSG_ERROR_FOLDER_SUMMARY_MISSING due to FileSize==ZERO in nsMsgDatabase::CheckForErrors is cause, How is the error detected and processed upon Virtual Folder open?

If such problem upon Virtual Folder open, why rebuild-index won't occur upon all Virtual Foldeer open, even though "EndOfFile: 1601/01/01 9:00:00" looks to be saved in \Spam.msf always?

Comment 135

[WADA:World Anti-bad-Duping Agency]

FYI.
Value in file timestamp field in Query log by Process Monitor is UTC when SMB file, and is perhaps "64-bit value representing the number of 100-nanosecond intervals since January 1, 1601 00:00:00(UTC)".
In my environment, TIMEZONE is GMT+09:00. So, "64bits ZERO in EndOfFile: field" is perhaps shown by Process Monitor as "EndOfFile: 1601/01/01 9:00:00". Shown value perhaps depends on TIMEZONE, and it's perhaps different from mine in your environment.

Comment 136

[WADA:World Anti-bad-Duping Agency]

Matthew Jurgens(bug opener), can you get very small log for following?
(0) Create pretty small folders, call FolderA and FolderB.
    Copy a pretty small mail to FolderA and FolderB, and view mails.
    Add Received column to FolderA and FolderB. (to force .msf update)
    Create a Virtual Folder with search target=FolderA and FolderB. Call VF-X.
    Click other than VF-X, FolderA, FolderB, in order not to open these folders
    after restart.
    Confirm that folders are healthy, and terminate Tb.
    If required, restart Tb several times.
(1) With Process Monitor started, with NSPR logging enabled, restart Tb.
    Other than VF-X, FolderA, FolderB is selected at folder pane.
(2) Send 3 small mails which are moved by filter to FolderA,
    and send other 3 mails which are moved by filter to FolderB, and Get Msgs.
    Mails are downloaded to Inbox and are moved to FolderA and FolderB by filter.
(3) Click FolderA and view mail. (FolderA is explicitly opened after filter move)
(4) Click VF-X and view mail. (FolderB is not explicitly opened after filter move)
(5) View all mails in VF-X.
(6) Terminate Tb. Keep back up of NSPR log. Terminate Process Monitor.
(7) Keep backup of FolderA/FolderA.msf/FolderB/FolderB.msf.
Note: Because Offset/Length is written in Process Monitor log., what data Tb wrote to .msf file can be checked.

Comment 137

[WADA:World Anti-bad-Duping Agency]

Attachment: Logic flow of Virtual Folder open

Quick summary of nsMsgXFVirtualFolderDBView::OnNewSearch process.

> (1) nsMsgXFVirtualFolderDBView::OnNewSearch calls GetMsgDatabase() for each scope(=searchtTarget folder).
> (2) nsMsgLocalMailFolder::GetMsgDatabase calls GetDatabaseWOReparse()
> (3) nsMsgLocalMailFolder::GetDatabaseWOReparse calls OpenDatabase()
> (4) nsMsgLocalMailFolder::OpenDatabase calls OpenFolderDB()
> (5) nsMsgDBService::OpenFolderDB calls Open()
> (6) nsMsgDatabase::Open calls nsMsgDatabase::OpenInternal()
> (7) nsMsgDatabase::OpenInternal writes "nsMsgDatabase::Open(...)" log,
>     and calls OpenMDB(),
>     and writes "error opening db ..." log if OpenMDB() fails,
>     then calls CheckForErrors() and returns ReturnCode from CheckForErrors().
> (7-1) nsMsgDatabase::OpenMDB physically opens <FolderName>.msf file.
> (7-2) nsMsgDatabase::CheckForErrors sets summaryFileExists=false if fileSize==0.
>          1267   summaryFileExists = exists && fileSize > 0;
>       And returns NS_MSG_ERROR_FOLDER_SUMMARY_MISSING if summaryFileExists=false.
>          1323   return (summaryFileExists) ? err : NS_MSG_ERROR_FOLDER_SUMMARY_MISSING;

If "EndOfFile: 1601/01/01 9:00:00"==ZERO and "NS_MSG_ERROR_FOLDER_SUMMARY_MISSING from CheckForErrors()" and "existence of new message" is only cause, I think rebuild-index should always occur if SMB1 is used.
So, I guess that "NS_MSG_ERROR_FOLDER_SUMMARY_MISSING from CheckForErrors()" is usually ignorered but is processed if other coditions are met, or "NS_MSG_ERROR_FOLDER_SUMMARY_MISSING from CheckForErrors()" is replaced or superceeded by normal status if some other conditions are met.

Comment 138

[WADA:World Anti-bad-Duping Agency]

Attachment: Close and Open at same time : \Spam.msf and \Spam log at 7:22:46 in 14.Sent.Spam.PML

Open of \Spam.msf file just after Close of \Spam.msf file is seen at 7:22:46.

> 2189  7:22:46.5421079  Spam.msf  IRP_MJ_CLEANUP SUCCESS                          
> This is End of close of \Spam.msf file by OS for Tb's Task #1
> -------------------------------------------------------------------
> Next access to \Spam.msf starts.
> 2190  7:22:46.5837989  Spam.msf  Read Metadata IRP_MJ_DIRECTORY_CONTROL SUCCESS             
>       Type: QueryDirectory, Filter: Spam.msf, 2: Spam.msf
> This is by preparation work for open of \Spam.msf in Tb's Task #2,
> or by open \Spam.msf request from Tb's Task #2.
> --------------------------------------------------------------------
> Close operation of \Spam.mfs at Tb's Task #1 ends, and Tb's Task #1 writes log.
> 2013-10-11 22:22:46.588000 UTC - 0[e0f140]: closing database
>   S:dataMozilla ThunderbirdProfilesMatthewMailLocal FoldersSpam.msf
> --------------------------------------------------------------------
> OS processed open request of \Spam.msf from Tb's Task #2
> 2191  7:22:46.5941377  Spam.msf  IRP_MJ_CREATE SUCCESS
>       Desired Access: Generic Read/Write, Disposition: Open, Options: Synchronous IO Non-Alert,
>       Non-Directory File, Attributes: N, ShareMode: Read, Write,
>       AllocationSize: n/a, OpenResult: Opened 
> After file open and end of entire \Spam.msf file read, following is logged by Tb's Task #2.
> 2013-10-11 22:22:46.603000 UTC - 0[e0f140]:
>   nsMsgDatabase::Open(S:dataMozilla ThunderbirdProfilesMatthewMailLocal FoldersSpam.msf, FALSE, 727c410, TRUE)

This may be following.
  Task #1                             Task #2
  Global variable X=true is set
    while processing \Spam.msf.
  Tb requests close \Spam.msf to OS.  Task #2 starts.
  OS ends close \Spam.msf process     Tb knows Global variable X==true. 
                                      Tb requests open \Spam.msf to OS.
                                      Because \Spam.msf is closed,
                                      OS starts "open \Spam.msf" process.
  "Close \Spam.msf" call ends.        "Open \Spam.msf" ends normally.
  Tb writes "closing data base" log.  Tb starts to read entire \Spam.msf data.
  Tb continues cleanup works.
    for examle,
    reset Global variable X to false.
  Task #1 ends all required process.
                                      After end of entire \Spam.msf file read,
                                      Tb sets other variable Y to true,
                                      because X was true.

If so, problem may be;
  After it, Virtual Folder is opened.
  Because of Y==true, NS_MSG_ERROR_FOLDER_SUMMARY_MISSING from CheckForErrors
  is processed, and rebuild-index is invoked.

Comment 139

[WADA:World Anti-bad-Duping Agency]

A suspect.
If time between "1038 db->Close(true);" and "1041 rv = msgDBService->OpenFolderDB(this, false, ..." is sufficiently long, no problem.
If "1041 rv = msgDBService->OpenFolderDB(this, false, ..." is processed by Tb and OS very quickly after end of "1038 db->Close(true);" call, something wrong may occur.
"db->Close(true)" and "msgDBService->OpenFolderDB" may be executed under different tasks.

> http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsLocalMailFolder.cpp#1013
> 1013 nsresult nsMsgLocalMailFolder::OpenDatabase()
>(snip)
> 1022   rv = msgDBService->OpenFolderDB(this, true, getter_AddRefs(mDatabase));
> 1023     if (rv == NS_MSG_ERROR_FOLDER_SUMMARY_MISSING)
> 1024     {
> 1025     // check if we're a real folder by looking at the parent folder.
> 1026     nsCOMPtr<nsIMsgFolder> parent;
> 1027     GetParent(getter_AddRefs(parent));
> 1028     if (parent)
> 1029     {
> 1030       // This little dance creates an empty .msf file and then checks
> 1031       // if the db is valid - this works if the folder is empty, which
> 1032       // we don't have a direct way of checking.
> 1033       nsCOMPtr<nsIMsgDatabase> db;
> 1034       rv = msgDBService->CreateNewDB(this, getter_AddRefs(db));
> 1035       if (db)
> 1036       {
> 1037         UpdateSummaryTotals(true);
> 1038         db->Close(true);
> 1039         mDatabase = nullptr;
> 1040         db = nullptr;
> 1041         rv = msgDBService->OpenFolderDB(this, false,
> 1042                                         getter_AddRefs(mDatabase));
> 1043         if (NS_FAILED(rv))
> 1044           mDatabase = nullptr;
> 1045       }
> 1046     }

Comment 140

[Matthew Jurgens]

(In reply to WADA from comment #136)
> Matthew Jurgens(bug opener), can you get very small log for following?

I have done the following (times are approximate - should be plus or minus about 1 second).
This test included 3 TB opens with logs. The third TB open downloaded 4 emails. No folder rebuilds occurred.
Note that you can get all logs and MSF data in a single download from
http://www.edcint.co.nz/LargeFiles/Completed/17.Test_NoRebuilds.All.zip

TB OPEN #1
==========
Open TB with Inbox selected. No mails to download.
7:39:20 Click TestFolder A
7:39:30 Click TestFolder B
7:39:40 Click Virtual Folder which is a search on TestFolderA and TestFolderB
7:39:50 Click Inbox
7:40:00 Close TB
Created the following files:
http://www.edcint.co.nz/LargeFiles/Completed/17.Test_NoRebuilds.1.PML.zip
http://www.edcint.co.nz/LargeFiles/Completed/17.Test_NoRebuilds.1.TB.log


TB OPEN #2
==========
Open TB with Inbox selected. No mails to download.
7:44:00 Click TestFolder A
7:44:10 Click TestFolder B
7:44:20 Click Virtual Folder which is a search on TestFolderA and TestFolderB
7:44:30 Click Inbox
7:44:42 Click Get Mail
7:44:45 (approx) Close TB
Created the following files:
http://www.edcint.co.nz/LargeFiles/Completed/17.Test_NoRebuilds.2.PML.zip
http://www.edcint.co.nz/LargeFiles/Completed/17.Test_NoRebuilds.2.TB.log


TB OPEN #3
==========
From a Linux command line send 2 mails which will be placed into TestFolderA and 2 that will be placed into TestFolderB.
Open TB with Inbox selected. 4 mails to download.
7:47:30-35 (approx) Mail starts to download
7:47:50 Click TestFolder A
7:48:00 Click TestFolder B
7:48:10 Click Virtual Folder which is a search on TestFolderA and TestFolderB
7:48:20 Click Inbox
7:44:30 Close TB
Created the following files:
http://www.edcint.co.nz/LargeFiles/Completed/17.Test_NoRebuilds.3.PML.zip
http://www.edcint.co.nz/LargeFiles/Completed/17.Test_NoRebuilds.3.TB.log
http://www.edcint.co.nz/LargeFiles/Completed/17.Test_NoRebuilds.TestFolderA.msf
http://www.edcint.co.nz/LargeFiles/Completed/17.Test_NoRebuilds.TestFolderB.msf

Comment 141

[WADA:World Anti-bad-Duping Agency]

Thanks for small log for normal case with detailed test procedure.
I can check difference between normal case and Spam/Sent cases in log set #14.

Similar "Open and immediate Close" or "Close and immediate Re-open" is also seen on Sent folder case in 14.Sent.Spam.PML/14.Sent.Spam.TB.log.
And, after it, rebuild-index is invoked on Sent.
However, phenomenon doesn't look identical, even though it's similar.

Virtual Folder is opened at 22:21:32.
> 2013-10-11 22:21:32.291000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\ ... \edcint.co.nz\Search to Rebuild Folders.msf, FALSE, ee1380, TRUE)

(1) \Sent.msf is opened at 7:22:01.009
> 2013-10-11 22:22:01.009000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\ ... \edcint.co.nz\Sent.msf, FALSE, ee21c0, TRUE)

(2) "Append mail data to \Sent file" is executed at 7:22:01.33
>            "Time of Day"    ,"Path"                               ,"Category","Operation"   ,"Result" ,"Detail"
>            "7:22:01.3310949","S:\Data\ ... \edcint.co.nz\Sent"    ,"Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset: 95,116,578, Length:  8,014"
>            "7:22:01.3381851","S:\Data\ ... \edcint.co.nz\Sent"    ,"Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset: 95,113,216, Length: 12,288, I/O Flags: Non-cached, Paging

(3) After append of mail data to \Sent, \Sent.msf is shown in "NN open DB's" log,
    although MM in "MM drs in use" is reduced during 22 second.
> 2013-10-11 22:22:02.619000 UTC - 0[e0f140]: S:\ ... \edcint.co.nz\Sent.msf - 242 hdrs in use
> 2013-10-11 22:22:05.041000 UTC - 0[e0f140]: S:\ ... \edcint.co.nz\Sent.msf - 241 hdrs in use
> 2013-10-11 22:22:24.166000 UTC - 0[e0f140]: S:\ ... \edcint.co.nz\Sent.msf - 1 hdrs in use

(4) \Sent.msf is closed at 22:22:40 (opened at 7:22:01.009)
> 2013-10-11 22:22:40.994000 UTC - 0[e0f140]: closing database S:\ ... \edcint.co.nz\Sent.msf

(5) \Sent.msf is opened again after 8 seconds, at 22:22:48.791 (previously closed at 22:22:40),
    and is immediately closed at 22:22:48.916(after 125 msec)
> 2013-10-11 22:22:48.791000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\ ... \edcint.co.nz\Sent.msf, FALSE, 727e550, TRUE)
> 2013-10-11 22:22:48.916000 UTC - 0[e0f140]: closing database S:\ ... \edcint.co.nz\Sent.msf

(6) At 22:22:57, rebuild-index is invoked.
(6-1) At 22:22:57, "open, close immediatel, re-open immediately, then log for ...\Temp\Mozilla Mailnews\Sent.msf" happens.
> 2013-10-11 22:22:57.072000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\ ... \edcint.co.nz\Sent.msf, FALSE, 727c410, TRUE)
> 2013-10-11 22:22:57.228000 UTC - 0[e0f140]: closing database S:\ ... \edcint.co.nz\Sent.msf
> 2013-10-11 22:22:57.306000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\ ... \edcint.co.nz\Sent.msf, FALSE, 727c410, TRUE)
> 2013-10-11 22:22:57.384000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\DOCUME~1\mjurgensLOCALS~1\Temp\Mozilla Mailnews\Sent.msf, FALSE, 727c670, TRUE)
> 2013-10-11 22:22:57.384000 UTC - 0[e0f140]: error opening db 80550006
(6-2) At 22:23:03, \Sent.msf is closed.
> 2013-10-11 22:23:03.838000 UTC - 0[e0f140]: closing database S:\ ... \edcint.co.nz\Sent.msf

Virtual Folder is closed at22:26:41
> 2013-10-11 22:26:41.728000 UTC - 0[e0f140]: closing database S:\ ... \edcint.co.nz\Search to Rebuild Folders.msf

Virtual Folder is opened again at 22:26:42
> 2013-10-11 22:26:42.166000 UTC - 0[e0f140]: nsMsgDatabase::Open(S:\ ... \edcint.co.nz\Search to Rebuild Folders.msf, FALSE, ee0c60, TRUE)

Virtual Folder is opened again, immediately, at 22:26:42.
> 2013-10-11 22:26:42.338000 UTC - 0[e0f140]: closing database S:\ ... \edcint.co.nz\Search to Rebuild Folders.msf

In Sent folder case, trigger doesn't look "Virtual Folder open", although bug opener says that rebuild-index occurred when Virtual Folder was opened.
"Cause of rebuild-index" seems "Open MsgDatabase followed by immediate close" which happened at 22:22:48.
  Upon the immediate close, flag of "rebuild-index is needed" is set.
  Then rebuild-index is invoked upon nect MsgDatabase open.
If Virtual Folder is relevant, it may be issue like following.
  - Virtual Folder already opens \Sent.msf when an event happens on \Sent.
  - Virtual Folder is listner of state change of \Sent.msf and/or \Sent.
    "Handler of an event on Folder" and "Virtual Folder" interferes each other.

Comment 142

[Matthew Jurgens]

(In reply to WADA from comment #141)
Surely there is enough information for someone to start looking at the TB code now?
Looking at the code now given what you have found may be able to lead directly to a fix.

Comment 143

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #142)
> Surely there is enough information for someone to start looking at the TB
> code now?
I don't think sufficient. I believe addtional analysis of Process Monitor log is needed.
- There is funny log.
    7:21:54.9782819 S:\ ... \Local Folders\Spam.msf
    Read Metadata IRP_MJ_QUERY_INFORMATION SUCCESS
    Type: QueryStandardInformationFile,
    AllocationSize: 1,048,576,
    EndOfFile:        119,514,
    NumberOfLinks: 1, DeletePending: False, Directory: False
  Because "read of entire Spam.msf by Tb" is executed on 119,514 bytes only,
  EndOfFile: is correct. Why "AllocationSize: 1,048,576"?
- Open or Close log of MSDB:5 is written by only one module.
  Tb's request to OS before it, and Tb's request to OS by other than the module,
  is known only by Process Monitor log analyis.
But, I believe it's sufficient for starting analysis by Debug build.
  Is FileSize=0 returned from OS? Is FileSize=0 saved in .msf?
  Is m_parsefolder set to true?
  To know it, debug build will be perhaps needed.

Comment 144

[WADA:World Anti-bad-Duping Agency]

Quick summary of open/close of Virtual Folder, Spam, and Sent.
If "reduced MM hdr in use of Sent" is "delete mails by retention policy", and if auto-compact is enabled, problem may be following.
  because of FileSize==ZERO, Campact sets m_parsefoldre=true,
  then Rebuild-Index is executed upon access by Virtual Folder.
However, if so, why no log by MsgPurge:5 on Sent folder?
In Spam case, cause of problem may be following "little dance" in nsMsgLocalMailFolder::OpenDatabase().
  msgDBService->OpenFolderDB()
  because of NS_MSG_ERROR_FOLDER_SUMMARY_MISSING,
  => msgDBService->CreateNewDB() for existent DB
  => db->Close(true)
  => msgDBService->OpenFolderDB() again by same task
  => mDatabase = nullptr; => DB will be closed sooner or later

(1) Virtual Folder is opened
> 22:21:32.291000 UTC : nsMsgDatabase::Open(S:\ ... \Search to Rebuild Folders.msf, FALSE, ee1380, TRUE)

(2-1) Spam is opened to append mail data.
      Perhaps by filter move, because "write to Inbox, write to Spam, truncate Inbox is logged.
> 22:21:54.947000 UTC : nsMsgDatabase::Open(S:\ ... \Local Folders\Spam.msf, FALSE, 727e090, TRUE)
(2-2) "Append mail data to \Sent file" at 7:21:55
> "Time of Day"    ,"Path",                                 "Category","Operation"   ,"Result" ,"Detail"
> "7:21:55.1912890"."S:\Data\ ... \Local Folders\Spam",     "Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset: 4,429,449, Length: 5,862","2892","340","File System"
> "7:21:55.1940361"."S:\Data\ ... \Local Folders\Spam",     "Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset: 4,427,776, Length: 8,192, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O","2892","340","File System"
(2-3) Close Spam after filter move
> 22:21:56.306000 UTC : closing database S:\ ... \Local Folders\Spam.msf

(3-1) Sent is opened, and a mail is written.
      Mail send was done?
> 22:22:01.009000 UTC : nsMsgDatabase::Open(S:\ ... \edcint.co.nz\Sent.msf, FALSE, ee21c0, TRUE)
(3-2) "Append mail data to \Sent file" at 7:22:01.33
> "Time of Day"    ,"Path"                               ,"Category","Operation"   ,"Result" ,"Detail"
> "7:22:01.3310949","S:\Data\ ... \edcint.co.nz\Sent"    ,"Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset: 95,116,578, Length:  8,014"
> "7:22:01.3381851","S:\Data\ ... \edcint.co.nz\Sent"    ,"Write"   ,"IRP_MJ_WRITE","SUCCESS","Offset: 95,113,216, Length: 12,288, I/O Flags: Non-cached, Paging
(3-3) Sent is not closed after append of mail data

(4-1) At 6:22:02, Spam is opened.
> 22:22:02.588000 UTC : nsMsgDatabase::Open(S:\ ... \Local Folders\Spam.msf, FALSE, 4fb2550, TRUE)
(4-2) At same time, "hdrs in use" number of Sent is reduced.
> 22:22:02.619000 UTC : S:\ ... \edcint.co.nz\Sent.msf - 242 hdrs in use
> 22:22:02.666000 UTC : closing database S:\ ... \Local Folders\Spam.msf
> 22:22:05.009000 UTC : nsMsgDatabase::Open(S:\ ... \Local Folders\Spam.msf, FALSE, ee2090, TRUE)
> 22:22:05.041000 UTC : S:\ ... \edcint.co.nz\Sent.msf - 241 hdrs in use
> 22:22:05.072000 UTC : closing database S:\ ... \Local Folders\Spam.msf
> 22:22:24.166000 UTC : S:\ ... \edcint.co.nz\Sent.msf - 1 hdrs in use

(5) After 16 seconds, Sent is closed.
> 22:22:40.994000 UTC : closing database S:\ ... \edcint.co.nz\Sent.msf

(6-1) "Open Spam, Close Spam immediately" is repeated twice by 727c410
> 22:22:46.463000 UTC : nsMsgDatabase::Open(S:\ ... \Local Folders\Spam.msf, FALSE, 727c410, TRUE)
> 22:22:46.588000 UTC : closing database S:\ ... \Local Folders\Spam.msf
> 22:22:46.603000 UTC : nsMsgDatabase::Open(S:\ ... \Local Folders\Spam.msf, FALSE, 727c410, TRUE)
> 22:22:46.697000 UTC : closing database S:\ ... \Local Folders\Spam.msf
(6-2) "Open Spam, Close Spam immediately" is execute immediately by 727e550
> 22:22:48.791000 UTC : nsMsgDatabase::Open(S:\ ... \edcint.co.nz\Sent.msf, FALSE, 727e550, TRUE)
> 22:22:48.916000 UTC : closing database S:\ ... \edcint.co.nz\Sent.msf

(7) Rebuild-Index is inovked on Spam and Sent.
    Virtual Folder is perhaps accessed again.
(7-1) At 22:22:56, rebuild index of Spam starts.
> 22:22:56.463000 UTC : nsMsgDatabase::Open(S:\ ... \Local Folders\Spam.msf, FALSE, 8fe68e0, TRUE)
> 22:22:56.572000 UTC : closing database S:\ ... \Local Folders\Spam.msf
> 22:22:56.681000 UTC : nsMsgDatabase::Open(S:\ ... \Local Folders\Spam.msf, FALSE, 8fe68e0, TRUE)
> 22:22:56.759000 UTC : nsMsgDatabase::Open(C:DOCUME~1mjurgensLOCALS~1TempMozillaMailnewsSpam.msf, FALSE, 8fe6a10, TRUE)
> 22:22:56.759000 UTC : error opening db 80550006
(7-2) At 22:22:57, rebuild index of Sent starts.
> 22:22:57.072000 UTC : nsMsgDatabase::Open(S:\ ... \edcint.co.nz\Sent.msf, FALSE, 727c410, TRUE)
> 22:22:57.181000 UTC : S:\ ... \Local Folders\Spam.msf - 5 hdrs in use
> 22:22:57.228000 UTC : closing database S:\ ... \edcint.co.nz\Sent.msf
> 22:22:57.306000 UTC : nsMsgDatabase::Open(S:\ ... \edcint.co.nz\Sent.msf, FALSE, 727c410, TRUE)
> 22:22:57.353000 UTC : S:\ ... \Local Folders\Spam.msf - 5 hdrs in use
> 22:22:57.384000 UTC : nsMsgDatabase::Open(C:DOCUME~1mjurgensLOCALS~1TempMozillaMailnewsSent.msf, FALSE, 727c670, TRUE)
> 22:22:57.384000 UTC : error opening db 80550006
> 22:22:57.416000 UTC : S:\ ... \Local Folders\Spam.msf - 5 hdrs in use

(8) Spam and Sent is closed at same time
> 22:23:03.838000 UTC : closing database S:\ ... \Local Folders\Spam.msf
> 22:23:03.838000 UTC : closing database S:\ ... \edcint.co.nz\Sent.msf


(9) Ater 3 minutes, Spam is re-opened, and closed immediately.
> 22:26:03.025000 UTC : nsMsgDatabase::Open(S:\ ... \Local Folders\Spam.msf, FALSE, 90fd380, TRUE)
> 22:26:03.228000 UTC : closing database S:\ ... \Local Folders\Spam.msf

(10) Virtual Folder is closed.
    This is by shutdown, because last log is 22:26:42.353000
> 22:26:41.728000 UTC : closing database S:\ ... \Search to Rebuild Folders.msf
> 22:26:42.166000 UTC : nsMsgDatabase::Open(S:\ ... \Search to Rebuild Folders.msf, FALSE, ee0c60, TRUE)
> 22:26:42.338000 UTC : closing database S:\ ... \Search to Rebuild Folders.msf

(11) Last line of log.
> 22:26:42.353000 UTC : closing database S:\ ... \pop3.edcint.co.nz\Inbox.msf

Matthew Jurgens(bug opener):
Is auto-compact enabled? If yes, is Compact actually executed on folders? 
(No Procss Monitor log for "nstmp" was found in log set #14. So, Compact was not executed while you were getting log.)

Comment 145

[Matthew Jurgens]

(In reply to WADA from comment #144)
> Is auto-compact enabled? If yes, is Compact actually executed on folders? 
Auto-compact is enabled but did not execute during the test from Comment #140.

(In reply to WADA from comment #143)
>   To know it, debug build will be perhaps needed.
If you get me a debug build I will run it.

Comment 146

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #145)
> Auto-compact is enabled but did not execute during the test from Comment #140.
I asked about "auto-compact" in daily use.
Is file size of Inbox, Sent, Spam etc.(not .msf) automatically/silently reduced in daily use?
(No log for Compact in log set #14. "write to nstmp or nstmp-N" should be logged by Process Monitor if Compact happens.)

> If you get me a debug build I will run it.
I can't create nor get debug build. Help by developers is needed.

Comment 147

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #140)
> I have done the following (times are approximate - should be plus or minus　about 1 second).

Thanks for small logs again.
However, difference between TestFolderA and TestFolderB looks "number of existent mails in folder" only...
Purpose of my requested procedure was to know difference in log between;
- Test-Folder-#1
  After append mail data by filter, explicit folder open(by folder click),
  then access via Viryual Folder.
- Test-Folder-#2
  After append mail data by filter, access via Virtual Folder without explict folder open.
  (almost same as \Spam case in log set #14)
Other concern is :
  When Virtual Folder is opened, search target folder is ;
    opened(cached), or not opened(not cached)
  Several patterns for this are perhaps seen in log taken by you.

Anyway, small log for TestFolderA will be pretty useful, because you provided log for 3 different patterns. Even if "difference from Spam/Sent case in log set #14" is no "open/close immediately * 2 by same task" only or no "single open/close immediately by someone" only, log for normal case is useful.

Can you attach small TestFolderA and TestFolderA.msf after shutdown of Tb?

(no need of TestFolderB and TestFolderB, because difference looks number of mails/file size only)
Because I requested you to test with "folder of a few small mails only" + "several filter moved small mails only", the files should be pretty small.

By the way.

Have you tried to translate "MSDB5:5 log and Process Monitor log" to "your test scenario"?
MsgDatabase is automatically closed after an inactive period(opened again when activity happens), and is kept open for a while after close request from a task(cached for a while, around 5 minutes by default, then physically closed). And, Process log merely says "data of Length: bytes is written or read at Offset:".
So, it's neally impossible to know your actions from log only. This is reason why I asked you to write "when you did what or wat happened".

14.Sent.Spam.PML / 14.Sent.Spam.TB.log actually contains evidences of problem of this bug.
And following are fortunately discovered.
- what does MSDG:5 log for C:\ ... \Temp\MozillaMailnews\XXX.msf mean
- funny "EndOfFile: 1601/01/01 9:00:00"
- some Tb logics around FileSize=ZERO and NS_MSG_ERROR_FOLDER_SUMMARY_MISSING due to it
- "little dance" in a module of Tb
However, this bug is already hard to read, because it took many comments to know above very important things.
After summariznig them, with getting sufficient reference data with pretty small folder(s), with adding "when 14.Sent.Spam.PML / 14.Sent.Spam.TB.log was taken by you, what was done or occurred at when", I think "new bug open" is better for us.

Matthew Jurgens, what do you think?

Comment 148

[WADA:World Anti-bad-Duping Agency]

Oh sorry, you already provided .msf file data. Because "two mail are moved" is guaranteed by your statements, no need of TestFolderA and TestFolderB. Sorry, again.

Comment 149

[WADA:World Anti-bad-Duping Agency]

Following is Addon for Tb to check data in xpconnect wrapped object which I created for testing Thunderbird.
> http://www.h2.dion.ne.jp/~radon/mozilla/tb-addon/WinBackMyTrash-0-1-0-001.xpi
(1) Save above xpi file to HDD.
    If you rename it to .zip and UNZIP it, you can see all source code of the addon.
    Please check source if you can not trust me :-)
(2) Install the addon to Tb.
    Via Customize of Toolbar, add Button-2("i" mark) to MenuBar or ToolBar.
(3) Open Error Console, and Clear
(4) Select TestFolderA at folder pane, select a mail at thread pane.
(5) Select "Selected Folder Info" -> "Selected Folder" at drop down menu　of button-2.
(6) Find following log in Error Console, Copy to Text Editor, and check value of following properties.
> msgFolder.msgDatabase.dBFolderInfo : [xpconnect wrapped nsIDBFolderInfo]
>   msgFolder.msgDatabase.dBFolderInfo / Property
>     folderDate = 1382597741
>     folderSize = 5558
Matthew Jurgens, what value is set?

Note:
This addon is for test purpose only. Although this addon does only print object's property value to Error Console, except some exceptions, this addon may produce memory leak, may interfere other addons or Tb, may cause crash. After check, please uninstall immeadiately.

Comment 150

[Matthew Jurgens]

(In reply to WADA from comment #147)
> I think "new bug open" is better for us.
Create a new one if you think it is best. 
How do we get someone to look at the code now?

(In reply to WADA from comment #149)
> Matthew Jurgens, what value is set?

2013/10/24 10:16:19.735 GMT WinBackMyTrash : Button_2,click#=1,req=13,label=Selected Folder
msgFolder.msgDatabase.dBFolderInfo : [xpconnect wrapped nsIDBFolderInfo]
  msgFolder.msgDatabase.dBFolderInfo / Property
    characterSet = ISO-8859-1
    characterSetOverride = false
    effectiveCharacterSet = ISO-8859-1
    expiredMark = 0
    expungedBytes = 0
    flags = 4
    folderDate = 1382518054
    folderName = 
    folderSize = 92594367
    highWater = 92575296
    imapTotalPendingMessages = 0
    imapUidValidity = -1
    imapUnreadPendingMessages = 0
    knownArtsSet = 
    locale = 
    mailboxName = TestFolderA
    numMessages = 293
    numUnreadMessages = 0
    sortOrder = 2
    sortType = 18
    version = 1
    viewFlags = 0
    viewType = 0(In reply to WADA from comment #147)

> (In reply to Matthew Jurgens from comment #140)
> However, difference between TestFolderA and TestFolderB looks "number of
> existent mails in folder" only...
TestFolderA has 293 mails, TestFolderb has 6 emails

Comment 151

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #150)
>    folderDate  = 1382518054
>    folderSize  = 92594367
>    mailboxName = TestFolderA
>    numMessages = 293

Following is logged in 17.Test_NoRebuilds.3.PML when two mails are appended to end of file by filter move.
> 00:01:22.9857566 17:47:33.1278672 thunderbird.exe S:\ ... \Local Folders\TestFolderA
>   Read Metadata IRP_MJ_QUERY_INFORMATION SUCCESS
>   Type: QueryStandardInformationFile, AllocationSize: 93,323,264, EndOfFile: 92,588,917, NumberOfLinks: 1, DeletePending: False, Directory: False
> 00:01:22.9861378 17:47:33.1282484 thunderbird.exe S:\ ... \Local Folders\TestFolderA
>   Write IRP_MJ_WRITE SUCCESS
>   Offset: 92,588,917, Length: 2,725
> 
> 00:01:24.1258818 17:47:34.2679924 thunderbird.exe S:\ ... \Local Folders\TestFolderA
>   Read Metadata IRP_MJ_QUERY_INFORMATION SUCCESS
>   Type: QueryStandardInformationFile, AllocationSize: 93,323,264, EndOfFile: 92,591,642, NumberOfLinks: 1, DeletePending: False, Directory: False
> 00:01:24.1264699 17:47:34.2685805 thunderbird.exe S:\ ... \Local Folders\TestFolderA
>   Write IRP_MJ_WRITE SUCCESS
>   Offset: 92,591,642, Length: 2,725
Windows API for File I/O used by Tb doesn't look to return "EndOfFile: 1601/01/01 9:00:00" in FASTIO_QUERY_INFORMATION directly.

Question is: 
  Why rebuild-index was invoked upn Virtual Folder open when 14.Sent.Spam.PML was taken.
(1) How, Why, by What logic, with What conditions, was rebuild-index invoked on Span and Sent when folder was accessed via Virtual Folder.
(2) When, How, by Whom, was the condition produced in \Spam.msf and \Sent.msf?

As I wrote before, if intentional outdated-msf condition by manual Mbox file size/timestamp change, even if "error opening db 0x80550005" is writen in MSGDB:5 log by following code(0x80550005==NS_MSG_ERROR_FOLDER_SUMMARY_OUT_OF_DATE), 
  nsMsgDatabase::OpenInternal
    nsresult rv = OpenMDB(summaryFilePath.get(), aCreate, sync);
    if (NS_FAILED(rv))
    PR_LOG(DBLog, PR_LOG_ALWAYS, ("error opening db %lx", rv));
rebuild-index was not automatically invoked.
i.e. nsMsgLocalMailFolder::GetDatabaseWOReparse is used.

Rebuild-index was automatically invoked upon "explicit folder open" and "view folder property". "Explicit folder open" and "view folder property" perhaps uses nsMsgLocalMailFolder::GetDatabaseWithReparse which will perhaps call ParseFolder() who executes rebuild-index.

As seen in MSGDB:5 log, Virtual Folder is always opened just after restart, as done on Inbox of all acounts and special folders such as Trash, Sent, Spam.
Because "open database Tb log for search target folders" is not seen just after restart of Tb, "open of search target folders" is not done upon restart of Tb.
However, "event listner" may be set for search target folders by Virtal Folder open. If so status change in search target folder may be sent to Virtual Folder. If so, Virtual Folder may access search target folder upon "status change of search target folder".

Because first MsgPurge activity is kicked after 5 minites from restart as seen in Tb log, open/close by MsgPurge is irrelevant to Spam/Sent case in 14.Sent.Spam.PML.
So, I'm suspecting ;
  "Virtual Folder" and "MsgDatabase close after filter move, after inactive period etc."
  may interfere each other.
And, AFAIK, new mails in Inbox, Sent is notified to Unified Folder(==Virtual Folder)  named Inbox and Sent, in order to resolve following issue which is called "BUG" by users.
  As known by name of "Search Folder", current Virtual Folder is "Search Folder"
  which shows search result of multiple search target folders as an "Message Folder".
  So, new message count of Unified folder is not updated automatically, and new message
  is not shown in Unified Folder, until user re-opens Unified Folder
  by "click other folder, then click the Unified Folder".

Comment 152

[WADA:World Anti-bad-Duping Agency]

Attachment: 17.Test_NoRebuilds.3.TB.log-Upon-SHOTDOWN-Folder-Not-Closed-Log.TXT

(In reply to Matthew Jurgens from comment #140)
> TB OPEN #3
> ==========
> From a Linux command line send 2 mails which will be placed into TestFolderA
> and 2 that will be placed into TestFolderB.
> Open TB with Inbox selected. 4 mails to download.
> 7:47:30-35 (approx) Mail starts to download
> 7:47:50 Click TestFolder A
> 7:48:00 Click TestFolder B
> 7:48:10 Click Virtual Folder which is a search on TestFolderA and TestFolderB
> 7:48:20 Click Inbox
> 7:44:30 Close TB
> Created the following files:
> http://www.edcint.co.nz/LargeFiles/Completed/17.Test_NoRebuilds.3.PML.zip
> http://www.edcint.co.nz/LargeFiles/Completed/17.Test_NoRebuilds.3.TB.log
> http://www.edcint.co.nz/LargeFiles/Completed/17.Test_NoRebuilds.TestFolderA.
> msf
> http://www.edcint.co.nz/LargeFiles/Completed/17.Test_NoRebuilds.TestFolderB.
> msf

There is no "closing database" log for following three folders.
  \Local Folders\TestFolderA.msf
  \Local Folders\TestFolderB.msf
  \edcint.co.nz\Inbox.msf
This kind of phenomenon may be a cause of "Outdated-msf" condition or "summay folder missing" condition generated during shutdown of Tb.
(If so, "error opening db ..." should be logged upon first open of \Spam.msf / \Sent.msf in 14.Sent.Spam.PML. 14.Sent.Spam.PML case is not "Outdated-msf" condition or "summay folder missing" generated during shutdown.)

Did you kept backup of TB log file after normal end of shutdown of Tb?
Or you kept backup while Tb is running without "sync" parameter?

Comment 153

[Matthew Jurgens]

(In reply to WADA from comment #152)
> Did you kept backup of TB log file after normal end of shutdown of Tb?
No

> Or you kept backup while Tb is running without "sync" parameter?
I don't know what you mean

Comment 154

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #153)
> (In reply to WADA from comment #152)
> > Did you kept backup of TB log file after normal end of shutdown of Tb?
> No

You wrote as follows...
> 7:44:30 Close TB
> Created the following files:
> http://www.edcint.co.nz/LargeFiles/Completed/17.Test_NoRebuilds.3.TB.log

> > Or you kept backup while Tb is running without "sync" parameter?
> I don't know what you mean

NSPR log data is bufferred. So, if NSPR log file is copied while Tb/Fx is running, log data still held in buffer is not copied. "NSPR_LOG_MODULES=...,sync,..." parameter is to disable the "buffering of NSPR log data".

Anyway, please provide non-confusing data for problem analysis always.

Comment 155

[Matthew Jurgens]

(In reply to WADA from comment #154)
> > Created the following files:
> > http://www.edcint.co.nz/LargeFiles/Completed/17.Test_NoRebuilds.3.TB.log
If this is the file you want it is still available at that URL

> NSPR log data is bufferred. So, if NSPR log file is copied while Tb/Fx is
> running, log data still held in buffer is not copied.
> "NSPR_LOG_MODULES=...,sync,..." parameter is to disable the "buffering of
> NSPR log data".
I always copy the log file after TB is closed

Comment 156

[Matthew Jurgens]

Any progress?

Comment 157

[WADA:World Anti-bad-Duping Agency]

> Any progress?
Nothing.
Because no "error opening DB" is logged for Sent.msf nor Spam.msf but "error opening DB" for Backup Database is suddenly logged for Sent and Spam(indicator of Rebuild Index is being executed for Sent and Spam) after several ope.close request of Sent.msf or Spam.msf, it looks for me that .msf of Sent and Spam in 14.Sent.Spam.TB.log is already broken before restart of Tb :-)
If the Rebuild-Index is invoked just after "append of downloaded mail data to Sent file or Spam file", I can suspect something wrong during MsgDB update for mail add/delete. But Rebuild-Index is invoked far after from "download of mail and append data to Sent file or Spam file.

Because "activity on S:\data\ ... \edcint.co.nz\Search Test Folders.msf just after appending downloaded mail data to Sent file and Spam file" is seen in TB.log and PML, I'm suspecting interfere between "activity on mail folder just after adding downloaded mail" and "actvity in Virtual Folder by notification of new mail in Search Target folder". "Unified Folder" is a "Search Folder==Virtual Folder", and Sent, Spam is usually "Search Target folder of Unified folder".
So, it may be interfere among "download to Mbox" and "access to Mbox by Virtual Folder #1" and "access to Mbox by Virtual Folder #2", or problem when simulteneous close/open by "download to Mbox" and "access to Mbox by Virtual Folder #1" and "access to Mbox by Virtual Folder #2".

    Close action at task #1                    Open action at task #2(Virtual Folder)
               |
      After end of MsgDB use for download,       Addition of new mail to Mbox is notified.
        close of MsgDB is sceduled
        because "time of no activity"
        exceeded a threshold.
      Update Mbox.msf file content               To access Mbox,
      Close file named Mbox.msf                    Open of MsgDB is scheduled
      File named Mbox.msf is closed
        at OS/File API level,
      but at Tb level,
        close of MsgDB is still in progress.

      Lock of resource is released.
                                                 Because lock is released,
                                                 Open of MsgDB process starts.
      close process for MsgDB continues          Open file named Mbox.msf, and read content
                                                 Before all control blocks is
                                                   consistently updated by task #1,
                                                   task #2 updates control blocks 
      All control bloks are updated
      Close of MsgDB completes


No additional report from you?
How freequesntly did your problem happen?
You could find procedure to see problem consistently?

Comment 158

[WADA:World Anti-bad-Duping Agency]

Because "offset/length of written data to Mbox.msf file" is logged by Process Monitor, if all of following when problem occurred is available, "what is written to Mbox.msf" can be known, although it's partially.
  Tb log, Process Monitor log,
  periodical copy of Mbox.msf file during Tb is runnning
Do you have such data?

Comment 159

[WADA:World Anti-bad-Duping Agency]

Matthew Jurgens(bug opener), do you sill frequently see your problem in recent Tb 24?
Do you see phenomenon of bug 931303 in Tb 24?

As I wrote in bug 931303 comment #10(see also bug 931303 comment #8), return code of "MsgDBHdr add" is done in "filter move" code from Tb 19. After it, error message is shown if the return code !=0, then user are aware of problem during filter move.

Some users look experiencing bug 931303 frequently even after repeated rebuild-index. This is pretty similar to this bug.
It looks for me;
  This bug occurs in many Tb user's environment, but Tb user wasn't aware of this bug
  because "filter move" silently generated "error in MsgDB" until Tb 17.

I'm suspecting "new mail download -> notification to Unified Folder -> contention between filter move and Unified Folder code".
- Tb 17 : No RC check, then "filter move" appends mail data to message store
          without proper MsgDB update, thus error is generated in MsgDB.
- Tb 24 : RC is checked, so error message is shown when "already generated error" exists.
          Even after rebuild-index, such contention occurs.
          Because RC is checked, error message is shown when such contention happens.

Comment 160

[WADA:World Anti-bad-Duping Agency]

If you already have hidden "smart mailboxes" account for Unified Folder, do you see your problem frequently after "remove smart mailboxes account and Unified Folder"?

(1) Remove accountN / serverX for Unified Folders from prefs.js
  - mail.accountmanager.accounts = account1, ... , accountN, ...
  - mail.account.accountN.server = serverX
  - mail.server.serverX.hidden   = true
    mail.server.serverX.hostname = smart mailboxes
    mail.server.serverX.name     = Unified Folders
    mail.server.serverX.type     = none
    mail.server.serverX.userName = nobody
(2) Delete mail directory for Unified Folders which is pointed by following entry.
  - mail.server.serverX.directory-rel =  [ProfD]Mail/smart mailboxes
    mail.server.serverX.directory     = ...\...\Mail\smart mailboxes
(3) Delete Unified Folder definitions which has following uri in virtualFolders.dat .
      uri=mailbox://nobody@smart%20mailboxes/...
          where ... = Archives, Drafts, Inbox, Junk, Sent, Trash etc.
(4) Delete panacea.dat (folder cache which contains folder name, file path)
(5) Restart Tb.
    If "unresponsive script" warning is shown, reply "continue".
    Never reply "stop script execution".
    After it, never enter "View/Folders/Unified".

Comment 161

[Matthew Jurgens]

Attachment: nounifed.png

This partial screenshot shows the visible TB folder structure after following the instructions in Comment #160

Comment 162

[Matthew Jurgens]

(In reply to WADA from comment #160)
> you see your problem frequently after "remove smart mailboxes account and
> Unified Folder"?
I followed these instructions and it appeared to be much better for a few days (not much activity though) but now after about a week I am sure that the situation has not really improved.

I have just uploaded https://bugzilla.mozilla.org/attachment.cgi?id=8356470 which shows what my folder structure looks like after following the instructions.

Please verify that this is what I should expect to see.

Comment 163

[WADA:World Anti-bad-Duping Agency]

Attachment: Extracted data(with Write type PML only) from 14.Sent.Spam.TB.log and 14.Sent.Spam.PML

Attached is a summary of your 14.Sent.Spam case.
POP3 download and filter move is(Not "After Classification" in your case);
  Append downloaded mail data to file named Inbox.
  If "filter move" happens,
    Copy the appended mail data to MoveTarget folder.
    Truncate file named Inbox at file size before "append to Inbox by download".
    (after it, "Inbox is same as before download")
So, "Write data to Sent, Spam" for which rebuld-index happened was extracted, and "Write data to Trash" was extracted for comparison, and "Truncate at Inbox" was extracted from PML log file.
For ease of reading, Tb log for relevant folders only are extracted.
For ease of reading, TB log(timestamp is UTC) anf PML log(original timestamp in log data is UTC, but Data/time in report is local time where report is generated, GMT+0900 in my environment) is merged based on UTC(GMT time).

You can see following by "repeated search for string of 'Inbox'".
a. Only one mail was moved from Inbox to Spam  => rebuild-index happened
b. Only one mail was moved from Inbox to Sent  => rebuild-index happened
c. Only one mail was moved from Inbox to Trash => rebuild-index does NOT occur

Even though "mail adding to Spam/Sent" was "filter move of one mail only" in your 14.Sent.Spam case, rebuild-index happened.
Common observed phenomenon in both Spam case and Sent case is;
- Immediate folder open just after folder close(Close msf after inactive period)
- Immediate close of folder just after open(perhaps "dance" in comment of code)
So, if "access by Virtual folder" is not key event in your problem, I suspect "MsgDatabase close after an idle time" which was inroduced from Tb 3.

What is your "new mail check interval" setting?
If 5 minutes, "filter move target folder close" may occur just around next new mail download time.
> mail.db.idle_limit = 300000
> This is in milliseconds, so closed after 300 seconds==after 5 minutes.
> http://mxr.mozilla.org/comm-central/source/mailnews/base/util/msgDBCacheManager.js#108
> 108     let idleLimit = Services.prefs.getIntPref("mail.db.idle_limit");
> 109     let maxOpenDBs = Services.prefs.getIntPref("mail.db.max_open");
> 110 
> 111     let closeThreshold = Date.now() - idleLimit;

Will frequency of your problem reduced by "larger than new mail check interval" and "prime number minutes" in mail.db.idle_limit?
   7 minutes :  420000 msec
  11 minutes :  660000 msec
  13 minutes :  780000 msec
  17 minutes : 1020000 msec

Comment 164

[WADA:World Anti-bad-Duping Agency]

Sorry, wrong pasting.
c. 8 mails were moved from Inbox to Trash => rebuild-index does NOT occur

Comment 165

[Matthew Jurgens]

> What is your "new mail check interval" setting?
One account is 1 minute, one account is 5 minutes, two other accounts are 10 minutes

> Will frequency of your problem reduced by "larger than new mail check
> interval" and "prime number minutes" in mail.db.idle_limit?
>    7 minutes :  420000 msec
>   11 minutes :  660000 msec
>   13 minutes :  780000 msec
>   17 minutes : 1020000 msec
mail.db.idle_limit has been set to 420000

Additionally, I have reverted the changes from comment #160
I am also now running v27.0 from the Beta channel.

Comment 166

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #165)
> > What is your "new mail check interval" setting?
> One account is 1 minute, one account is 5 minutes, two other accounts are 10 minutes

"mail.db.idle_limit=3000000" may produce "Filter move target folder close" just around "future new mail download" if "1 minute" or "5 minutes".
Do you see frequent Rebuild-index on "filter move target folder" of account of new mail check interval="10 minutes"?

Because "1 minutes" is used in your case,, "prime number" * "prime number seconds larger/less than 60" * 1000 may be better.
  "7 * 67 * 1000" for "near 7 minutes",  "17 * 53 * 1000" for "near 17 minutes"

Comment 167

[Matthew Jurgens]

> Do you see frequent Rebuild-index on "filter move target folder" of account
> of new mail check interval="10 minutes"?
I don't think so. I think most of the rebuilds are on the account with a 1 minute check frequency. However, this account is also the most active so the 1 minute check might be a red herring.

> Because "1 minutes" is used in your case,, "prime number" * "prime number
> seconds larger/less than 60" * 1000 may be better.
>   "7 * 67 * 1000" for "near 7 minutes",  "17 * 53 * 1000" for "near 17
> minutes"
I have see the idle time to 500,000

Comment 168

[kev]

We have to do a bit more testing with our busiest Tb users, but I wanted to report that on Sernet Samba 4.1.3 and Tb 24.2.0, we seem to be no longer experiencing this bug, but I'm not sure exactly which version switch helped, because we've been working around it by having all our mail locally stored for our busiest Tb users.  I'd be interested to hear if Matthew Jurgens has the same bug on Samba 4.1.x or not.

Comment 169

[Matthew Jurgens]

> I have set the idle time to 500,000
I have been running with this for nearly a week now and there is no change to the problem.

Comment 170

[Matthew Jurgens]

(In reply to kev from comment #168)
> because we've been working around it by having all our mail locally stored
> for our busiest Tb users.  
I suspect that this is probably had the biggest impact

> I'd be interested to hear if Matthew Jurgens has
> the same bug on Samba 4.1.x or not.
I am on Samba 4.0.8 and can't get to Samba 4.1 until 
1) it is released for Fedora 19 (this may not happen)
2) or I upgrade to Fedora 20 (this might take me a while)

I'd be interested to see if you put your mail back on Samba for for your busiest users.

Comment 171

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #169)
> > I have set the idle time to 500,000
> I have been running with this for nearly a week now and there is no change to the problem.
Still same frequency like "one rebuild-index per a day on at least one folder"?
Still same frequency like "at least one rebuild-index per a day on same number of folders"?

Because new message is checked every 60 sec in your case, "close by idle time and open by filter move at same time" may happen every 500sec * 3 times = 1500sec = 60sec(1 min) * 25 times.
This is a reason why I asked you to check with "prime number seconds * 1000" miliseconds as idle limit.

Comment 172

[WADA:World Anti-bad-Duping Agency]

(In reply to kev from comment #168)
> Sernet Samba 4.1.3 and Tb 24.2.0, we seem to be no longer experiencing this bug,
> but I'm not sure exactly which version switch helped, (snip)

SMB1? Windows client?

> https://www.samba.org/samba/history/samba-4.1.0.html
> https://www.samba.org/samba/history/samba-4.1.3.html
Samba 4.1 looks for improvements in SMB2 and enancements in SMB3, although fix in samba-4.1.0 or security patch in samba-4.1.1 to 4.1.3 may resolve problem in SMB1 too, because Tb's remote mail folder access may be similar to "attack" for Samba.
Please note that this bug is for SMB1 with Win-XP SMB client who supports SMB1 only.
Problem with SMB2 or SMB3 may be different, even if symptom of ".msf is broken" or "rebuild-index frequently happens" is same.

Comment 173

[Matthew Jurgens]

(In reply to WADA from comment #171)
> Still same frequency like "one rebuild-index per a day on at least one
> folder"?
> Still same frequency like "at least one rebuild-index per a day on same
> number of folders"?
> 
It seems about similar. I have now changed the 1 minute check frequency to 4 minutes.

Comment 174

[WADA:World Anti-bad-Duping Agency]

Attachment: Extracted data(wth Read/Write PML of Sent/Spam) from 14.Sent.Spam.TB.log and 14.Sent.Spam.PML

Previous one was summary with Write type PML log only for Span/Sent folder.
This summary is with both Read & Write type PML log for Spam.msf/Spam and Sent.msf/Sent.

Read/Write of Spam.msf/Spam and Sent.msf/Sent is pretty normal.
If SMB file share only problem, it may be following.

As I already wrote, following IRP_MJ_QUERY_INFORMATION is returned from OS.
> \Local Folders\Spam  Read Metadata IRP_MJ_QUERY_INFORMATION  SUCCESS
>    Type: QueryNetworkOpenInformationFile,
>    CreationTime:  2013/09/25 6:40:32, LastAccessTime: 2013/10/12 4:05:34,
>    LastWriteTime: 2013/10/12 7:21:55, ChangeTime:     2013/10/12 7:21:55,
>    AllocationSize: 1601/01/01 9:00:00, EndOfFile: 1601/01/01 9:00:00,
>    FileAttributes: A
> \edcint.co.nz\Sent  Read Metadata  IRP_MJ_QUERY_INFORMATION  SUCCESS
>    Type: QueryNetworkOpenInformationFile,
>    CreationTime:  2013/10/12 7:22:06, LastAccessTime: 2013/10/12 7:22:22,
>    LastWriteTime: 2013/10/12 7:22:06, ChangeTime:     2013/10/12 7:22:06,
>    AllocationSize: 1601/01/01 9:00:09, EndOfFile: 1601/01/01 9:00:09,
>    FileAttributes: A

Because AllocationSize/EndOfFile == 0 is always returned from OS, ZERO is written in .msf file as "last file size of msgStore file".
Because AllocationSize/EndOfFile == 0 is consistently returned from OS upon next msgDatabase open, and because timestamp is always correct and consistent, "outdated msf condition" won't occur.
Because "write mail data to Spam or Sent file by messegae filter" is done by write request of "append data at EndOfFile" type, there is no problem in mail data writing/reading.

However, when "dance" happens, additional check is executed by Tb, and Tb finds "msgStore file size=0" in .msf file data.

Comment 175

[WADA:World Anti-bad-Duping Agency]

Following is my add-on for testing of Tb.
> http://www.h2.dion.ne.jp/~radon/mozilla/tb-addon/WinBackMyTrash-0-1-0-005-MsgDBHdrAddDelete.html
"Folder Info of current folder" of "Button-1", or "WBMT: Selected Folders Info" of folder context menu, is for dumping folder related data to Error Console.
Some size/timestamp related properties of msgFolder/msgDatabase etc. is dumped by addon(from 0-1-0-005, for this bug). I don't know wheter such property is used for "outdated msf condition". Size/Timestamp for "outdated msf condition" may be held in string property.

Is size=0 seen in dump of folder data?

"Accessing msgDatabase object by add-on" invokes "DBOpened" event on folder. This indicates "Accessing msgDatabase by add-on" invokes "Message Database Open". This is perhaps similar to Tb's action invoked by "Folder property".

Is "Rebuild-index" invoked by dumping folder data?

"dump of folder data" supports "multiple folder selection at folder pane". So, you can check multiple folder's data at once.

Comment 176

[WADA:World Anti-bad-Duping Agency]

Major purpose of "button-1 -> MsgDB: Add Listner" of the add-on is logging of DBopen, msgDBHdr-add/delete, onAnnouncerGoingAway event etc. for ease of observation of this bug.
If "Repair Folder" is executed, onAnnouncerGoingAway event is kicked on the folder(msgDatabase).

Log on selected folders is automatically dumped to Error Console, only when;
  onEvent(msgDatabase event. DBOpened etc.) occurs
  or onItemEvent(folder event. DeleteOrMoveMsgCompleted etc.) occurs,
  and more than 100 lines are already logged when the event occurs. 
So, if you enable logging on folderA and folderB and do "move 10000 mails in folderA to folderB", following occurs.
  10000 msgDBHdrAdded   log lines on folderB is generaed in JavaScript array,
  10000 msgDBHdrDeleted log lines on folderA is generaed in JavaScript array,
  DeleteOrMoveMsgCompleted event(folder event) is kicked,
  then "20001 lines of log" is dumped to Error Console as an item of Error Console.
This is to avoid "time consuming dumping to Error Console" upon msgDBHdrAdded and msgDBHdrDeleted event.  
Please don't enable msgDatabase/msgFolder event logging of such heaviliy updated folders.
Please limit to some folders only(max 32 folders are accepted curretly), on which you frequently see this bug, such as Spam or Sent in 14.Sent.Spam.TB.log only.
I don't know "added event listener by add-on" will produce memory leak or Tb's conrol block corruption or not, if folder is renamed or deleted or moved. Please don't do rename/delete/move while you enable msgDatabase/msgFolder event logging.

Comment 177

[WADA:World Anti-bad-Duping Agency]

"Outdated msf condition" is detected by nsMsgBrkMBoxStore::IsSummaryFileValid.
> http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsMsgBrkMBoxStore.cpp#213
> IsSummaryFileValid uses GetFolderSize() and GetFolderDate()
> 241   folderInfo->GetFolderSize(&folderSize);
> 242   folderInfo->GetFolderDate(&folderDate);
> SetFolderSize()/GetFolderSize() and SetFolderDate()/GetFolderDate() is defied at here.
> http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsDBFolderInfo.cpp#413
>   SetUint64Property and kFolderSizeColumnName was used for folderSize.
>   SetUint32PropertyWithToken and m_folderDateColumnToken was used for folderDate.

idl for nsIDBFolderInfo is as follows.
> http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/public/nsIDBFolderInfo.idl#40
> 40   attribute unsigned long long folderSize;
> 41   attribute unsigned long folderDate;
> folderSize/folderDate is defined as; (setter/getter is used)
> http://mxr.mozilla.org/comm-central/ident?i=folderSize
>   mailnews/db/msgdb/src/nsDBFolderInfo.cpp (View Hg log or Hg annotations)
>     line 413, as member of class nsDBFolderInfo -- nsDBFolderInfo::GetFolderSize(uint64_t *size)
>     line 420, as member of class nsDBFolderInfo -- NS_IMETHODIMP nsDBFolderInfo::SetFolderSize(uint64_t size)
> http://mxr.mozilla.org/comm-central/ident?i=folderDate
>   mailnews/db/msgdb/src/nsDBFolderInfo.cpp (View Hg log or Hg annotations)
>     line 427, as member of class nsDBFolderInfo -- nsDBFolderInfo::GetFolderDate(uint32_t *folderDate)
>     line 434, as member of class nsDBFolderInfo -- NS_IMETHODIMP nsDBFolderInfo::SetFolderDate(uint32_t folderDate)

So, following data obtained by JavaScript via nsIDBFolderInfo.idl is same as data used by Tb's CPP code.
  msgFolder.msgDatabase.dBFolderInfo.folderSize
  msgFolder.msgDatabase.dBFolderInfo.folderDate

According to screenshot sent from you to me, following was shown in your environment.
  msgFolder.sizeOnDisk  = 351522951
  msgFolder.msgDatabase.dBFolderInfo.folderSize = 351522951
OS dosen't look to return "file size=ZERO in QueryNetworkOpenInformationFile" as "file size data for API call by Tb".
Developers of MS Windows perhaps knows pretty well about "SMB1 lies on file size" :-)

Because there is perhaps no problem in "filesize/timestamp data for outdated msf condition detection" in your case even when SMB1 is used, I guess problem around "dance" which may be produced by following.
- "close msgDatabse by idle time" and "open msgDatabase by filter move" at same time.
- timing hole when multiple tasks requested msgDatabase open when msDatabase is not opened.
  For example,
  while MsgDB caching process for msgDatabase open is in progress by task#1 and state is
  still cache-requested-but-not-completed-yet, msgDatabase open is requested by task#2.
If this kind of problem, "problem started to occur from Tb 3" can be explained, because "msgDatabase close by idle time threshold"(.msf file close while Tb is running) was introduced by Tb 3(till Tb 2, .msf file was closed only when shutdown of Tb), and because ".msf file open/read entire data from .msf file for msgDatabase open" and "writing back .msf file" takes far longer than local HDD file when network file is used.

Comment 178

[WADA:World Anti-bad-Duping Agency]

FYI.
> nsMsgLocalMailFolder::OpenDatabase() is caller of OpenFolderDB
> http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsLocalMailFolder.cpp#1010
>  1027       // This little dance creates an empty .msf file and then checks
>  1028       // if the db is valid - this works if the folder is empty, which
>  1029       // we don't have a direct way of checking.
> nsMsgDBService::OpenFolderDB is defined at;
> http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#92
>  109   if (cacheDB)
>  110   {
>  111     // this db could have ended up in the folder cache w/o an m_folder pointer via
>  112     // OpenMailDBFromFile. If so, take this chance to fix the folder.
>  (snip)
>  117     // if m_thumb is set, someone is asynchronously opening the db. But our
>  118     // caller wants to synchronously open it, so just do it.

Comment 179

[WADA:World Anti-bad-Duping Agency]

FYI.

When nsMsgDBService::OpenFolderDB is called, OpenFolderDB does do following.
(1) nsMsgDBService::OpenFolderDB calls msgDatabase->Open
http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#134
134   rv = msgDatabase->Open(summaryFilePath, false, aLeaveInvalidDB);
        msgDatabase->Open invokes nsMsgDatabase::OpenInternal
        => nsMsgDatabase::OpenInternal adds to DBcache
    http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#1240
           1240     AddToCache(this);
(2) nsMsgDBService::OpenFolderDB calls FinishDBOpen(), and return to caller.
http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#157
157   FinishDBOpen(aFolder, msgDatabase);
    nsMsgDBService::FinishDBOpen is defined at:
    http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#296
       FinishDBOpen() fills SyncCounts in msgDatabase
       after calling AllMsgHeadersTable->GetCount
         306  aMsgDB->m_mdbAllMsgHeadersTable->GetCount( ...
         310       aMsgDB->SyncCounts();
  Note: nsMsgDBService::AsyncOpenFolderDB() is also defined, but there is no caller of it.
        http://mxr.mozilla.org/comm-central/search?string=AsyncOpenFolderDB%28

Because there is time lag between following,
  (i)  nsMsgDatabase::OpenInternal  1240     AddToCache(this);
  (ii) nsMsgDBService::OpenFolderDB 157   FinishDBOpen()
(A-problem) if other task requests msgDatabase open of same msgDarabase between (i) and (ii), problem may happen. "MsgDB is cached, but SyncCounts is not updated yet" can occur.

IIRC, AllMsgHeadersTable->GetCount takes longer than we expect and bug for it is opened.
So, probability of (A-problem) may be "not so small", especially when file share.

If this kind of problem, "reducing number of mails in filter move target folder" can reduce frequency of problem.
If this kind of problem, report like following in bug 931303 may be explained.
  I reduced number of mails in filter move target folder and did Compact,
  After it, I don't experience problem.

Comment 180

[kev]

(In reply to Matthew Jurgens from comment #170)
> (In reply to kev from comment #168)
> > because we've been working around it by having all our mail locally stored
> > for our busiest Tb users.  
> I suspect that this is probably had the biggest impact

I was not clear:

1) we put our busiest users on local (problem obviously stopped)
2) we upgraded to 4.1.3
3) we put our busiest users back on Samba, and after a short amount of time we were not experiencing trouble
4) I posted saying 4.1.3 helped but we'd need more time to be sure

Unfortunately, I went a did a yum update to 4.1.4 without sufficient time, and somewhere in there the problem has begun again, and today was particularly bad.  So unless 4.1.4 re-broke something, you can disregard my earlier post.

> 
> > I'd be interested to hear if Matthew Jurgens has
> > the same bug on Samba 4.1.x or not.
> I am on Samba 4.0.8 and can't get to Samba 4.1 until 
> 1) it is released for Fedora 19 (this may not happen)
> 2) or I upgrade to Fedora 20 (this might take me a while)
> 
> I'd be interested to see if you put your mail back on Samba for for your
> busiest users.

Comment 181

[WADA:World Anti-bad-Duping Agency]

Correction.
  AddToCache() was not done at nsMsgDatabase::OpenInternal line 1240.
  It was when asynchronous open request which is not currently used.
When synchronous open request, AddToCache is done slightly later, after some checks such as "outdated msf condition". 
AddToCache() is executed by; 
  http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#1244
  nsMsgDatabase::OpenInternal calls CheckForErrors() and returns to caller.
    1244   return CheckForErrors(rv, true, summaryFile);
 => http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#1247
      nsMsgDatabase::CheckForErrors finally does do AddToCache() if synchronous open
      and if no error, then return to caller.
 at http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#1317
        1317   if (sync && (NS_SUCCEEDED(err) ||
                   err == NS_MSG_ERROR_FOLDER_SUMMARY_MISSING))
        1318     AddToCache(this);
        1319   return (summaryFileExists) ? err : NS_MSG_ERROR_FOLDER_SUMMARY_MISSING;

Comment 182

[WADA:World Anti-bad-Duping Agency]

FYI. I've opened bug 961931.

Comment 183

[Wayne Mery (:wsmwk)]

Are there any hints as to what patch caused this problem in tb3?

Comment 184

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #183)
> Are there any hints as to what patch caused this problem in tb3?

What is meaning of your question and purpose of your question?

Because I'm suspecting problem such as one which is wroe in my comment #177, I already wrote so.
And, root cause of such problems is needless to say that ".msf file close after some idle state" was introduced from Tb 3, if this bug was newly generated by Tb 3.

Please note that no one could find cause of problem of this bug so we are trying to find "what happens in bug opener's environment" continuously, for long time.

Comment 185

[Wayne Mery (:wsmwk)]

(In reply to WADA from comment #184)
> (In reply to Wayne Mery (:wsmwk) from comment #183)
> > Are there any hints as to what patch caused this problem in tb3?
> 
> What is meaning of your question and purpose of your question?

1. The purpose is to help identify the specific code, because no one attempted to identify the code by hunting with development builds. Indeed I missed the end of comment 177, where you specifically call out TB3 and confirming the reporter's information. Thanks for pointing out. (In bugs with hundreds of comments, it will be helpful to highlight key points in whiteboard.)
2. And so, if it helps, reporter(s) should be able to identify the specific TB3 development build which first shows this problem by regression testing

goal, naturally, is to narrow what source changes are needed to fix the issue?

Comment 186

[WADA:World Anti-bad-Duping Agency]

I did some tests around "outdated msf condition" again(see bug 917769 comment #51).
And, I found interesting phenomenon.
  - FolderX has "outdated msf condition".
    (file size/tmestamp is changed by Text Editor, by adding 
  - FolderX is not opened(FolderX.msf is not opened, msgDatabase is not cached)
  If property of msgFolder.msgDatabase of FolderX is accessed via XPCOM(getter is used)
  using JavaScript, exception with 0x80550005 ocurred as written in bug 917769 comment #51.
  In this case, "nsMsgDatabase::Open(C:\..." was not written by MSGDB:5 logging.
  Because my JavaScript code does nothing upon catch(e), rebuild-index is not invoked.
  After it, rebuild-index is executed by Virtual Folder or explicit folder open,
  and MSGDB:5 log of "nsMsgDatabase::Open(C:\...\Temp\MozillaMailnews\FolderX.msf" appears.

If Virtual Folder code accesses msgFolder.msgDatabase by JavaScript, and if Virtual Folder code hooks exception due to 0x80550005(outdated msf condition), and if Virtual Folder code invokes internal rebuld-index, no "nsMsgDatabase::Open for move target folder(Spam.msf or Sent.msf) without error opening db 80550005" may be explained.

i.e. "No error opening db 80550005 log by MSGDB:5 for Spam.msf/Sent.msf in your case" is not evidence that "outdated msf condition" doesn't occur in your case.

And, as seen in bug 917769 comment #51, "filter move" silently generate state like following when "outdated msf condition" already exists.
  - Mail data is appened to move target folder's msgStore file.
  - Moved mail is already deleted from "move source folder".
  - "Filter log" says "mail is normally moved by filter".
  - But "updated MsgDBHdr in msgDatabase" may be broken.
  - "Outdated msf condition" may be kept because ail data is appended without correct
    msgDatabase(.msf) update,
    or "Outdated msf condition" may be wrongly cleared, because first "outdated msf
    condition" which was detected upon start of "filter move" is ignored by "filter move".

So, problem may be following.
- Something wrong occurs(e.g. cause of bug 931303),
  then outdated msf condition is generated by "filter move",
  or corrupted msf data is generated by "filter move",
  thus, Virtual Folder code detected outdated msf condition, or Virtual Folder code
  detected something bad in .msf/msgDaabase, an invoked rebuild-index.
- The "something wrong" can occur frequently if SMB file share is used,
  because "slowness dure to file share" expands timing hole.
- Because you define Virtual Folder for many "filter move target folders",
  you can know event of "rebuld-index was invoked" easly and surely.
- Because "already generated outdated msf condition or sonething bad in .msf" is never
  cleard by "filter move" until user explicitly opens the folder or the folder is opened
  by Virtual Folder, "outdated msf condition(or something bad in msf)" is kept for long
  time in usual environment. So problem like bug 931303 can occur.

Comment 187

[WADA:World Anti-bad-Duping Agency]

Additional information.
Log of "db error opening db 80550005" is not written by MSGDB:5 if filter move, even when "outdated msf condition actually/already exists.
(0) Enable NSPR logging with MSGDB:5,MsgCopyService:5
(1) Filter move target folder=FolderX
    Edit file named FolderX, and appened mail data(Seen bit=Off, so Unread mail)
(2) Run filter. actual move target = AAA/ABC
> request 4d125c0 DoCopy - src  mailbox://hidden%40h.h.h@hidden.h.h.h/Inbox/Sub1A
>                          dest mailbox:// ... /AAA/ABC numItems 4 type=0
> nsMsgDatabase::Open(C:\ ... \Mail\pop.ops.dti.ne.jp\AAA.sbd\ABC.msf, FALSE, 7341080, TRUE)
> closing database    C:\ ... \Mail\pop.ops.dti.ne.jp\AAA.sbd\ABC.msf
>   Open/close is executed for each moved mail,
>   and finally closed.
> nsMsgDatabase::Open(C:\ ... Mail\pop.ops.dti.ne.jp\AAA.sbd\ABC.msf, FALSE, 7341080, TRUE)
> closing database    C:\ ... \Mail\pop.ops.dti.ne.jp\AAA.sbd\ABC.msf
> NotifyCompletion - src  mailbox://hidden%40h.h.h@hidden.h.h.h/Inbox/Sub1A
>                    dest mailbox:// ... /AAA/ABC
> request 4d125c0 Clearing OK request
>   - src  mailbox://hidden%40h.h.h@hidden.h.h.h/Inbox/Sub1A
>     dest mailbox:// ... /AAA/ABC numItems 4 type=0
(a) As seen in log, "db error opening db 80550005(or 80550006)" is not written,
    even though outdated maf condition actually exists.
    This is perhaps due to difference of "MsgDB Open request type when DoCopy" from
    "MsgDB open type when explicit folder open". 
(b) MsgDBHdrAdded   event is not invoked at move target folder
    MsgDBHdrDeleted event is invoked at move source folder
(3) Open Virtual Folder which has "move target folder" in search target folder.
Following log is written, and rebuild-index is executed on move target folder, then Unread count of the move target folder is updated at folder pane.
> nsMsgDatabase::Open(C:\DOCUME~1\wada\LOCALS~1\Temp\MozillaMailnews\ABC.msf, FALSE, 73f04b0, TRUE)
> error opening db 80550006

"Pattern of log" is pretty similar to Spam/Sent case of your 14.Sent.Spam.TB.log.

Question again.
Actually "no outdated msf condition before restart of Tb" when you obtained 14.Sent.Spam.TB.log?
Actually "no interfere by other software on Spam/Spam.msf or Sent/Sent.msf etc. on *file share server*" in your daily use?

No error code is shown when something wrong occured on msgDatabase by "filter move" nor by "Virual Folder open".
In contrast to it, "access to property of msgFolder.msgDatabase.xxx by JavaScript" produces exception with "return code to MsgDB open request" if outdated msf condition occurs, and the exception can be catched by try/catch(e).

I enhanced my add-on for this purpose.
  - Enable event listener for seleced folders(max 32). This is same as previous version.
  - View msgFolder property only of the selected folders. msgDatabase for the msgFolder
    is cached(already opened) or not-cached(not opened) can be known.
    By this operation, msgDatabase is not opened, so not-cached status is kept.
  - View msgFolder.msgDatabase property of the selected folders.
    If 0x80550005 or 0x80550006 happens, exception occurs, and is reported.
If you check such exception before forcing rebuild-index by Virtual Folder open, you can know reason why "rebuild-index" was invoked by Virtual Folder open, as long as message folder is not already opened(msgDatabase is not cached).

If you need this feature, let me know, please.

Comment 188

[WADA:World Anti-bad-Duping Agency]

If "outdated msf" condition is somehow generated, or if "fileSize=ZERO(null xxx.msf file)" condition is somehow generated, "mail move by message filter" can silently produce known problem of bug 261419 and bug 495760.
Once problem of bug 261419/bug 495760 happens, "Repair Folder"(Rebuil-Index) is needed.
Setting depeency to these bugs for ease of problem analysis, tracking, and search.

Comment 189

[Matthew Jurgens]

(In reply to WADA from comment #187)
> Question again.
> Actually "no outdated msf condition before restart of Tb" when you obtained
> 14.Sent.Spam.TB.log?
In that particular test, the folders were as clean and I could possibly make them ie all folders opened multiple times with no rebuild before TB closed. Refer to comment #110 for the exact procedure.

> Actually "no interfere by other software on Spam/Spam.msf or Sent/Sent.msf
> etc. on *file share server*" in your daily use?
Confirmed. No other software was accessing the files, not even a backup.

> I enhanced my add-on for this purpose.
> If you need this feature, let me know, please.
I will install your addon and run a test procedure again if this will be useful.

Comment 190

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #189)
> I will install your addon and run a test procedure again if this will be useful.
See following web page, Change history/Top Page linked in the page, please.
> http://www.h2.dion.ne.jp/~radon/mozilla/tb-addon/WinBackMyTrash-0-1-0-MsgDBHdrAddDelete.html
Following occur and logged?
- outdated-msf condition is detected upon msgDatabase object access by add-on
  (access via JavaScript), exception occurs and is hooked, ad is logged.
- "msgDatabase.summaryValid=false" which occurs if fileSize=0 occurs.

Comment 191

[WADA:World Anti-bad-Duping Agency]

FYI.
I updated my add-on(now 009).
msgDatabase.summaryValid can be checked automatically/periodically, and meesage is shown at Error Console only when error is detected by my add-on.
  Error :
    - msgDatabase.summaryValid = false. This occurs when .msf file size=ZERO happens.
    - exception happens by msgDatabase.summaryValid, if outdated msf condition exists.
Because add-on won't invoke Rebuild-Index when error is detected, "bad status of msgDatabase" is reported and is kept. This is difference from Virtual Folder's "check and silently force rebuld-index".

Comment 192

[WADA:World Anti-bad-Duping Agency]

Matthew Jurgens, see bug 495760 comment #7, please. That is test result with msgDatabase.summaryValid check by my add-on.
(a) If "outdated msf condition" already exists in .msf file before MsgDatabase open,
    rebuild-index is invoked by folder open which is requested by Virtual folder.
(b) If "msf file size=0" happens upon msgDatabase open which invokes "open/read .msf file",
    "open of msgDatabase" it self is successful and msgDatabase.summaryInvalid=true is set.
    Because msgDatabase is already cached, nothing occurs.
    Mail folder is normally accessed as folder of "number mail in folder=0".
    So, Virtual Folder can't force rebuild-index until folder is closed.   
    After close of the folder, if folder is opened by Virtual Folder open,
    "outdated msf condition" is raised, then rebuld-index is invoked by Virtual Folder.

Following can occur in current implementation.
a) "filter move" silently generates "outdated msf condition",
   or silently keeps "outdated msf condition",
b) "msf file size=0" status generates summayInvalid=true, and it will produce future
   "outdated msf condition".

A reason why rebuild-index is not invoked by "filter move" is "rebuild-index during filter move upn new mail download" is nearly impossible in current implementation. (see bug 261419, please)
In this case, Tb's desgin/implementation/code perhaps expects "rebuild-index upon next folder open", and mechanism of "rebuild-index upon next folder open" itself can be called "works pretty well".

Problem of this bug is;
  Why "unknown condition which will invoke rebuild-index" repeatedly/consistently occurs
  if SMB file share is used?
  What is the "unknown condition which will invoke rebuild-index" of this bug?

Comment 193

[WADA:World Anti-bad-Duping Agency]

FYI.
I updated my add-on(now 012). "msgDatabase.summaryValid of all POP3 folder & Local Folders" can be checked automatically/periodically. And, setTimeout(timeout=3sec) is used when Start is requested. So, you can start "automatic msgDatabase.summaryValid check of all POP3 folder & Local Folders" just after restart of Tb. by several mouse clickes only. What you need to do is; "Watch Error Console several times a day" only.

Comment 194

[WADA:World Anti-bad-Duping Agency]

Attachment: Extracted data(with Read/Write PML of Spam Only) from 14.Sent.Spam.TB.log and 14.Sent.Spam.PML

I checked 14.Sent.Spam.PML and 14.Sent.Spam.TB.log again, for \Mail\edcint.co.nz\Sent.msf(& Sent) only, and for \Mail\Local Folders\Spam.msf(& Spam) only, and interesting data is found.
  Spam.msf(& Spam) :
    1. \Spam is opened by filter move, Local Folders\Spam.msf is read, and updated, and Spam is closed.
    2. After next open of Spam, following is seen.
       In 14.Sent.Spam.PML, \Local Folders\Spam.msf is still being read, by DBOpen.
       However, in 14.Sent.Spam.TB.log, "Closeing DB log just after the DBOpen", "nsMsgDatabase::Open
       just after previous close", ..., is seen.
       i.e. Following occurs.
            # of "Read Spam.msf from Offset=0 to EOF by DBOpen + Write data to Span.msf by close"
         != # of " 'nsMsgDatabase::Open' + 'closing database' "
   3. Later, "full read of file named Spam" is seen in PML log,
      and "error opening db 80550006" is loged for ...\Temp\MozillaMailnews\Spam.msf,
      which is perhaps indicator of "rebuild-index".

In contrast to it, Sent case is different.
   1. nsMsgDatabase::Open is logged for Sent. This is DBOpen for filter move.
      Sent.msf is fully read, write data to Sent is logged, truncate at Inbox is logged,
      and data related to added message is written to Sent.msf,
   2. After it, no "closing database" occur for relatively long time.
      "Writeing small data at end of Sent.msf(22 bytes * 2 times) is seen several times in log.
   3. After a while, "closing database" occurs, and Sent.msf is updated.
   4. Later, Send is opened again, and logs which indicates "rebuild-index" are seen.
This is perhaps because Sent is search target folder of Unified Sent folder. Because Unified Sent accesses the Sent folder of an POP3 account, MsgDB close of Sent folder is not executed after filter move.

Phenomenon which is seen on \Mail\Local Folders\Spam.msf(& Spam) may be "Dance" which is written in comment of source code.

Comment 195

[WADA:World Anti-bad-Duping Agency]

In both Spam and Sent case, "updated data of XXX.msf after rebuild-index(perhaps)" is merely last part only.
i.e. "Data from Offset=0 to just before the last part of XXX.msf" is correct, even after "full read of file named XXX and rebuild-index operation".
This is pretty similar to following situation.
  Mail is appended to file named XXX by filter move.
  MsgDBHdr for added ail is created on memory.
  Before physical update of XXX.msf file, power failure happens.
However, if this case, "actual fileSize of XXX" != "fileSize of XXX saved in XXX.msf" always occurs, and "outdated msf condition" should be always raised upon nex folder open. And, if "outdated msf condition" already exists, and if Tb 19 or newer, bug 931303 should occur upon "filter move which is invoked by POP3 new mail download".

Problem like next?
  "Filter move" somehow fails to create "MsgDBHdr for added mail" correctly.
  Because MsgDB itself is available in memory, XXX,msf is closed normally,
  then "correct fileSize of XXX saved in XXX.msf" is correct;y written in XXX.msf vy close.
  However something wrong exists in MsgFBHdr for the added mail. So, MshFBHdr dile.
Please note that Spam & Sent case in 14.Sent.Spam.PML is "first filter move to a folder after restart of Tb upon POP3 mail download", and "mail.edcint.co.nz%3Amjurgens%40edcint.co.nz@gw/Sent case you sent to me" is also "first filter move to a folder after restart of Tb upon POP3 mail download".
  
Note:
Bug 782738 is for crash at startup. It may be crash at first mail download after startup.
As written in bug 931303, bug 782738 added RC check at step 2 of next.
  1. Folder open(MsgDB open) for filter move  <= still no care on "MsgDB Open error".
                                                 "outdated msf", "msf deleted" is ignored.
  2. Create MsgDBHdr for newly added mail     <= by bug 782738, RC is checked at here.
  3. Append mail data to msgStore file.
  4. Close folder(MsgDB).
And, if "outdated msf conditon" exists, error message of bug 931303 is shown at step 2.

Comment 196

[WADA:World Anti-bad-Duping Agency]

"MessageKey != MessageOffset if mail is moved by filter upon POP3 download" is relevant?

When fileSize of XXX is sss, "MessageKey and MessageOffset of mail which was moved to XXX by filter in single POP3 download" is :
                               MessageKey         MessageOffset 
  mail#(1),   size=SSS(1)   :  sss                sss
        |             |
  mail#(N),   size=SSS(N)   :  sss + SSS(N - 1)   MessageOffset_of_mail#(N-1) + SSS(N-1)
        |             |
  mail#(N+1), size=SSS(N*1) :  sss + SSS(N)       MessageOffset_of_mail#(N)   + SSS(N)

Some code may still assume "MessageKey == MessageOffset", especially in "Error handling code".
Repair Folder, Compact, Message Copy/Move, Filter Copy, Manual Filter Move, generates "MessageKey == MessageOffset".

Note:
"MessageKey != MessageOffset" is phenomenon from Tb 12. So, "this bug before Tb 12" can't be explained by "MessageKey != MessageOffset".

Comment 197

[WADA:World Anti-bad-Duping Agency]

Attachment: Extracted data(with PML of Sent Only) from 14.Sent.Spam.TB.log and 14.Sent.Spam.PML

Comment 198

[WADA:World Anti-bad-Duping Agency]

Attachment: Extracted data(with PML of Spam Only) from 14.Sent.Spam.TB.log and 14.Sent.Spam.PML

Comment 199

[WADA:World Anti-bad-Duping Agency]

Attachment: Extracted data(with PML of Sent Only) from 14.Sent.Spam.TB.log and 14.Sent.Spam.PML

MsgPurge:5 log is removed for ease of reading.

Comment 200

[WADA:World Anti-bad-Duping Agency]

Attachment: Extracted data(with PML of Spam Only) from 14.Sent.Spam.TB.log and 14.Sent.Spam.PML

MsgPurge:5 log is removed for ease of reading.

Comment 201

[WADA:World Anti-bad-Duping Agency]

"MessageKey != MessageOffset" is seen for many moved mails in HdrAdded/HdrDeleted event log sent to me. I can't think "MessageKey != MessageOffset" is cause of problem.
I can't find any "something bad" in PML log for update of msgStore file(Sent, Spam) and update of .msf file(Sent.msf, Spam.msf).
And, if "outdated msf condition" was generated, "outdated msf condition" should be logged by MsgDB:5 logging. However, "outdated msf condition" was not logged for Sent.msf and Spam.msf.

In "Rebuild-Index" event(we call so) of both Sent case and Spam case, "Full Read of .msf file(Sent.msf, Spam.nsf) for open MsgDB" and "Full Read of msgStore file(Sent, Spam)" are seen in PML log.
However, "physical update of .msf file" is (a) "a few thousands bytes data at end of msf file(call at Offfset=XXX)" + (b) several "short bytes" at end of .msf file" only.
"Existent data in .msf file from Offset=0 to Offset=XXX" is not modified by "Rebuild-Index" event(we call so).
(a) seems "corrected MsgDBHder for mail which was moved by filter".
(b) looks for me "ordinal data which is written upon MsgDB close(last used tmestamp etc.).
occurred.

When Spam, MsgDB was immediately closed after filter move upon POP3 mail download.
But when Sent, MsgDB was kept open after filter move upon POP3 mail download.
It looks for me "Reason why MsgDB was kept open" is;
 - Sent is Search Target folder of Unified Sent folder(a Virtual Folder/Search Folder)
 - Mail moved by filter in Sent, is read after 21 seconds by someone.
     Filter move:  07:22:01.3310949 \edcint.co.nz\Sent Write Offset: 95,116,578, Length: 8,014
     Mail read:    07:22:22.9358948 \edcint.co.nz\Sent Read  Offset: 95,116,578, Length: 8,014
   This indicates that someone accessed the moved mail == Someone accesses Sent folder.
When Sent, MsgDB is closed after a while.
And, when opened again(second Sent.msf open since restart of Tb) later, "Rebuild-Index" event(we call so) occurred.

So, in Sent case, I suspect prolem when "just after close MsgDB is requested, someone requested MsgDB open".

Spam case is different from Sent case.
1. After first MsgDB open and filter move to Spam, MsgDB is immeiately closed and Spam.msf is normally updated.
2. After a while, "Open MsgDB, then immediately close without Spam.msf file update" is seen several times.
3. Later, as in Sent, when MsgDB is opened, "Rebuild-Index" event(we call so) occurred.

In Spam case, I suspect "two open MsgDB requests at same time", or "while Tb is preparing MsgDB open for first request, second open is requested, and second open request interfered MsgDB open process". This may be "Dance" which is written in comment of source code.

Comment 202

[WADA:World Anti-bad-Duping Agency]

Attachment: Extracted data(with PML of Spam Only) from 14.Sent.Spam.TB.log and 14.Sent.Spam.PML

Sorry, wrong file uploaded. replacemet by correct one.

Comment 203

[WADA:World Anti-bad-Duping Agency]

Following is written data to Sent.msf, Spam.msf.
- All Write operation is "append to end of file".
  Nothing is removed from .msf file? .msf file size increases forever, until "Repair Folder"?
- "Added data by Rebuild-index(we call so)" == 2,599 bytes(Sent.msf), 3,223 bytes(Spam.msf)
  It looks that there is no problem in "Last added MsgDBHdr by filter move" .
  Update for Thread Column selection data etc. only?
  Note: "Two set of Thread column data" is sometimes seen in actual .msf file.
- After ".msf update by filter move and DB close", .msf file content is never modified
  even when repeated open/close-immediately/re-open-immediately is done by Tb.
  Corruption of data in MsgDB Object in memory?
  Corruption of data which is not read from .msf file?

> (A) Sent.msf
> 
> (A-1) Sent.msf, First open by filter move
>  * by filter move
>    07:22:01.4085874  \Sent.msf    FASTIO_WRITE   Offset: 455,413, Length:   291
>    07:22:01.4088091  \Sent.msf    FASTIO_WRITE   Offset: 455,704, Length:    41
>    07:22:01.4096870  \Sent.msf    FASTIO_WRITE   Offset: 455,745, Length:    24
>  * by some one
>    07:22:19.0289504  \Sent.msf    FASTIO_WRITE   Offset: 455,769, Length:    24
>  * by some one
>    07:22:26.1554226  \Sent.msf    FASTIO_WRITE   Offset: 455,793, Length:    24
>    07:22:26.1560142  \Sent.msf    FASTIO_WRITE   Offset: 455,817, Length:    24
>  * by close
>    07:22:34.6486371  \Sent.msf    FASTIO_WRITE   Offset: 455,841, Length:    24
>    07:22:34.6490061  \Sent.msf    FASTIO_WRITE   Offset: 455,865, Length:    24
>    07:22:36.0281345  \Sent.msf    FASTIO_WRITE   Offset: 455,889, Length:    63
> 
> (A-2) Sent.msf, Second(last) open for "Rebuild-Index
>  * by rebuild-index
>    07:23:00.8316523  \Sent.msf    FASTIO_WRITE   Offset: 455,952, Length: 2,599
>    07:23:00.8365133  \Sent.msf    FASTIO_WRITE   Offset: 458,551, Length:    24
>    07:23:00.8829936  \Sent.msf    FASTIO_WRITE   Offset: 458,575, Length:   104
>  * by close
>    07:23:03.8042540  \Sent.msf    FASTIO_WRITE   Offset: 458,679, Length:    24
> 
> (B) Spam.msf
> 
> (B-1) Spam.msf, First open by filter move
>  * by filter move
>    07:21:55.2564840  \Spam.msf    FASTIO_WRITE   Offset: 120,474, Length:   310
>    07:21:55.2566703  \Spam.msf    FASTIO_WRITE   Offset: 120,784, Length:    38
>    07:21:55.2573998  \Spam.msf    FASTIO_WRITE   Offset: 120,822, Length:    22
>  * by close
>    07:21:55.6267293  \Spam.msf    FASTIO_WRITE   Offset: 120,844, Length:    22
> 
> (B-2) Spam.msf, Last open for "Rebuild-Index
>  * by rebuild-index
>    07:22:56.9290643  \Spam.msf    FASTIO_WRITE   Offset: 120,866, Length: 3,223
>    07:22:56.9445658  \Spam.msf    FASTIO_WRITE   Offset: 124,089, Length:    22
>    07:22:57.0088678  \Spam.msf    FASTIO_WRITE   Offset: 124,111, Length:   124
>  * by close
>    07:23:03.8018289  \Spam.msf    FASTIO_WRITE   Offset: 124,235, Length:    22

Comment 204

[WADA:World Anti-bad-Duping Agency]

FYI.
Following may be evidence of "two MsgDB open requests at same time". This may be "last used timestamp".
> (A-1) Sent.msf, First open by filter move
>  * by some one
>    07:22:26.1554226  \Sent.msf    FASTIO_WRITE   Offset: 455,793, Length:    24
>    07:22:26.1560142  \Sent.msf    FASTIO_WRITE   Offset: 455,817, Length:    24
>  * by close(just before close?)
>    07:22:34.6486371  \Sent.msf    FASTIO_WRITE   Offset: 455,841, Length:    24
>    07:22:34.6490061  \Sent.msf    FASTIO_WRITE   Offset: 455,865, Length:    24
Because MsgDB is already cached(opened) in this case, physical .msf file open/read is not invoked, and "Cached MsgDB" is returned to DBOpen requester.
So, if this is evidence of "two MsgDB open requests at same time", and if "two MsgDB open requests at same time" is cause of problem("Spam.msf case" seems so), problem may be "serialization issue in Control Block update" between two requests. i.e. pretty popular "issue in multi-tasking" especially when multi-CPU.

Comment 205

[WADA:World Anti-bad-Duping Agency]

Following is diff of Trash.msf sent from bug opener to me.
  (1) Keep backup of Trash.msf (BeforeStartingTB)
  (2) Start Tb with MSGDB:5 logging enabled.
  (3) Open#1 : Trash is Normal opened, and normal close
  (4) Open#2 : Trash is opened to add mail to Trash.
      HdrAdded event is notified to AsgDB event listener. for only one mail.
      After a while, Trash is normally closed.
  (5) Open#3 : "Open, immediate close", is logged by MSGDB:5 once.
  (6) Open#4 : Opened again, and "rebuild-index(we call so) is invoked.
  (7) Terminate Tb, keep backup of Trash.msf (AfterTBClose)
This is almost same as Spam.msf case in 14.Sent.Spam.PML. 

> diff -ur TB-20140313184221.BeforeStartingTB/Matthew/[e-mail]/Trash.msf TB-20140313201057.AfterTBClose/Matthew/[e-mail]/Trash.msf
> --- TB-20140313184221.BeforeStartingTB/Matthew/[e-mail]/Trash.msf	2014-03-13 07:40:45.000000000 +1100
> +++ TB-20140313201057.AfterTBClose/Matthew/[e-mail]/Trash.msf	2014-03-13 20:10:50.000000000 +1100
> @@ -2069,3 +2069,17 @@
>  <(7C2=3350|[e-mail])>
>  [61E31D:^80(^B2^7C2)]
>  @$$}35}@
> +
> +@$${37{@
> +{-44B35F:^80 {(k^D5:c)(s=9)} }
> +@$$}37}@
> +
> +@$${38{@
> +@$$}38}@
> +
> +@$${39{@
> +{-44B35F:^80 {(k^D5:c)(s=9)} 61162F 61647A 618225 6195B9 61E31D 61FF3C }
> +@$$}39}@
> +
> +@$${3A{@
> +@$$}3A}@

That's all.
One MsgDBHdr is added, and .msf is closed. And, MsgDBHdr is correctly written by Open#2.
As known by 14.Sent.Spam.PML log, "Update of .msf" is correctly done afgter "filter move", and "update of .msf after rebuild-index(we call so)" is always "append some data at end of .msf" == "after offset of EOF upon full read of .msf for msgDatabase open".
In this case, Thread Column selection is already written in .msf, before above data, before restart of Tb. So, "Thread Column selection" data is not re-written to .msf by this Tb run.

I can't find any reason to invoke "Rebuild-Index", even though "outdated msf condition" nor ".msf file missing" is never raised by "MsgDatabase open".
"Verification of .msf when unknown condition, for safety" like action, instead of "Repair Folder"/"Rebuild-Index" like action?

Comment 206

[WADA:World Anti-bad-Duping Agency]

I wasn't aware of existence of getDatabaseWithReparse for JS...
I searched nsMsgLocalMailFolder::GetDatabaseWithReparse for CPP only...

> module who calls _indexerEnterFolder()
>     http://mxr.mozilla.org/comm-central/search?string=_indexerEnterFolder&find=&findi=&filter=^[^\0]*%24&hitlimit=&tree=comm-central
> (a) _worker_folderCompactionPass:
>     http://mxr.mozilla.org/comm-central/source/mailnews/db/gloda/modules/index_msg.js#1126
> (a) _worker_folderIndex:
>     http://mxr.mozilla.org/comm-central/source/mailnews/db/gloda/modules/index_msg.js#1324
> (c) _worker_messageIndex:
>     http://mxr.mozilla.org/comm-central/source/mailnews/db/gloda/modules/index_msg.js#1524
> 
> http://mxr.mozilla.org/comm-central/source/mailnews/db/gloda/modules/index_msg.js#611
> 592   _indexerEnterFolder: function gloda_index_indexerEnterFolder(aFolderID) {
> 610           this._indexingDatabase =
> 611             this._indexingFolder.getDatabaseWithReparse(null,null);
> =>
> http://mxr.mozilla.org/comm-central/source/mailnews/local/public/nsIMsgLocalMailFolder.idl#47
> 47   nsIMsgDatabase getDatabaseWithReparse(in nsIUrlListener aReparseUrlListener, in nsIMsgWindow aMsgWindow);
> This correspnds to;
> http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsLocalMailFolder.cpp#300
> 300 NS_IMETHODIMP nsMsgLocalMailFolder::GetDatabaseWithReparse(nsIUrlListener *aReparseUrlListener, nsIMsgWindow *aMsgWindow,nsIMsgDatabase **aMsgDatabase)

Gloda looks to invoke getDatabaseWithReparse always for indexing, in order to execute indexing even when "outdated msf condition" exists.
If "immediate close" like event or "open folder at same time" like event happens at same, upon Gloda requests getDatabaseWithReparse() because MsgDB is not opened yet or not cached yet, interfere of nsMsgLocalMailFolder::GetDatabaseWithReparse() may occur. This may generates "outdated msf condition" or "msf missing condition" which is not logged by MSDB:5(not ordinal return code to OpenFolder request).
If SMB file share is used, "full read of xxx.msf file for msgDatabase open" takes longer than local HDD.
  Note:
  Even when local HDD file, following was observed.
    Open folder -> DBOpen event is notified to event listener of my addon
    -> event listener checks(accesses) msgFolder.msgDatabse.summaryValid
    -> even though DBOpen event is already notified and msgFolder.msgDatabse.summaryValid value is
       returned, second DBOpen event is ntified o listener.
       This is perhaps "full read of XXX.msf is not completed yet when addon accessed summaryValid". 
"Detection of new mail -> Indexing by Gloda" may occur when folder is opened by Virtual Folder, if "folder has newly added mail" is already closed before Gloda starts indexing.

IIRC, Gloda is also feature from Tb 3, and is enabled by default.
Matthew Jurgens8bug opener), do you enable Gloda? (Global Search and Indexer)
Do you see your problem with Gloda disabled?

Note:
  Some code may still expect "msgDatabase is kept open, after he opened folder".
  So, if msgDabase is closed after he opened folder by "idle threshold time" or by "max open DB",
  some of them may behave unexpectedly.

Comment 207

[Matthew Jurgens]

(In reply to WADA from comment #206)
> IIRC, Gloda is also feature from Tb 3, and is enabled by default.
> Matthew Jurgens8bug opener), do you enable Gloda? (Global Search and Indexer)
> Do you see your problem with Gloda disabled?
> 
I have "Global Search and Indexer" enabled. I disabled it and the problem still occurs. I have re-enabled it.

Comment 208

[WADA:World Anti-bad-Duping Agency]

Problem when followin happens at same time?
(1) "Filer move" (1-a) opens XXX.msf via OpenWOReparse, (1-b) move message to XXX,
    and (1-c) close XXX.msf immediately.
(2) By (1-a), (2-a) itemAdded event is notified to folder listener of Virtual Folder.
    (2-b) Gloda tries to open MsgDB via OpenWithReparse.
(3) By (1-a), (3-a) itemAdded event is notified to folder listener of Virtual Folder(Search).
    (3-b) Virtual Folder(Search) tries to open MsgDB via OpenWithReparse.
Because (1-c), (2-b), (3-b) are three different event at different task after (1-b), these three events can occur at same time.

Spam.msf case.
  (1-a) moves one mail -> (1-b) -> immediate close by (1-c) -> imediate open by perhaps (2-b) or (2-c).
  After it, two "open then close immediately" is seen in log.
  Later, XXX.msf is opened by Virtual Folder, and log for "full read of XXX(msStore file)" and
  log for  "access to BacupDatabase" is observed.
Sent.msf case, Trash.msf case.
  (1-a)moves one mail -> (1-b).
  "Close XXX.msf by (1-c)" doesn't occur. MsgDB is kept open, and after a while, closed.  
  Sent.msf case  : No log of "open then close immediately". 
  Trash.msf case : One log of "open then close immediately". 
  Later, XXX.msf is opened by Virtual Folder, and log for "full read of XXX(msStore file)" and
  log for  "access to BacupDatabase" is observed.

Trash.msf, Sent.msf, Spam.msf is categorized as "largest msf file" in your environment.
(Archives\???.sbd\xxx.msf is larger, but "filter move" parhaps doesn't occur.)
Because SMB file share is used, "full read of XXX.msf"(==time required to parse folder) takes longer than local HDD.

In Trash.msf case, "msgDatabase.summaryValid=false just after DBOpened event notification" was not observed. msgDatabase.summaryValid=false is usually seen when "msf folder missing"(.msf is deleted) case.

In OpenWithReparse, NS_MSG_ERROR_FOLDER_SUMMARY_OUT_OF_DATE is not always "filSize in .msf != actual file size || timestamp difference is greater than timestamp_leeway". "Non-normal conditon" is notified to caller by "return code of NS_MSG_ERROR_FOLDER_SUMMARY_OUT_OF_DATE".

If DBOpen is internally done due to access like "msgFolder.msgDatabse.summaryValid value check by JavaScript code"(no OpenFolder request), MsgDB looks usually closed quickly by "idle time threshold".

It may be problem like following:
 "OpenWithReparse request while MsgDB is being closed", "Two OpenWithReparse at same time" can happen.
 Upon such condition, or contention happens, NS_MSG_ERROR_FOLDER_SUMMARY_OUT_OF_DATE is returned
 to caller of OpenWithReparse, or NS_MSG_ERROR_FOLDER_SUMMARY_OUT_OF_DATE is generated and processed
 during OpenWithReparse prcess.

Comment 209

[WADA:World Anti-bad-Duping Agency]

During POP3 download/filter move test with my addon, I found funny phenomenon.
(1) Log of passed MsgDBHdr data by "onHdrAdded" event listener.
    Upon OnHdrAdded event, some properties of msgDBHdr is logged by addon.
> DateTime=2014/03/18 22:17:33.546 GMT, EventType=MsgDBHdrAdded, 
>    MsgDB=mailbox://[e-mail]/Sent
>              messageKey = 8242,  messageOffset  =  1777,
It should be messageKey<=messageOffset always. Why messageKey>messageOffset?

(2)Actual msgDBHdr property value of mails in Sent folder after the download/filter move
> mail-1 = [messageKey] =    0  [messageOffset] =     0  [messageSize] =  567
> mail-2 = [messageKey] = 7675  [messageOffset] =  7675  [messageSize] =  565
> mail-3 = [messageKey] = 8240  [messageOffset] =  8240  [messageSize] =  622
> mail-4 = [messageKey] = 8241  [messageOffset] =  8862  [messageSize] = 1777
> mail-5 = [messageKey] = 8242  [messageOffset] = 10639  [messageSize] = 1777  <= downloaded/moved mail
(3) File Size of Inbox == 1,777 bytes. (data of "1 deleted mail" is held)

This "messageKey > messageOffset" is also seen in event listerner log sent from bug opener to me.
Contiuous "messageKey=(previous key + 1), messageOffset=0" on Trash is;
>    1. Inbox file size=0, 2. new mail is downloaded at Offset=0 of Inbox,
>    3. mail is moved to Trash by message filter, 4. Inbox is truncated at Offset=0
>    5. repeat 1 to 4.
Following is Trash.msf case you sent to me.
> DateTime=2014/03/13 07:45:37.950 GMT, EventType=MsgDBHdrAdded,
>   MessageKey=90136053, messageOffset=8968237,
>   MsgDB=mailbox://mail.edcint.co.nz%3Amjurgens%40edcint.co.nz@gw/Trash
>                         messageKey = 90136053
>   is far larger than messageOffset =  8968237 <= perhaps Offset in Inbox

"MsgDBHdr for filter moved message" is generated by destMailDB->CopyHdrFromExistingHdr.
If "msgDBHdr.messageOffset of passed msgDBHdr" is accessed by "onHdrAdded event listener" or by "OnItemAdded event listener", it's perhaps too early in some cases. "messageOffset value in Inbox" is obtained in some cases, if too quickly accessed after OnItemAdded/onHdrAdded event notification.

If someone starts "msgDBHdr handling of filter moved mail" by "onHdrAdded/OnItemAdded event notification", wrong messageOffset may be obtained. If mail data is retrieved from this "wrong Offset"(==Offset in Inbox, not modified to correct one yet), and if obtained mail data is unfortunately "broken data" for someone, someone may call "SetSummaryValid(false)". "SetSummaryValid(false)" will perhaps set "msgDatabase.summaryValud=false" and/or "msgDatabase.dBFolderInfo.version=0", in order to surely invoke rebuild-index upon next folder open. 

And, Gloda, gDBView etc. may start "msgDBHdr handling of filter moved mail" by "onHdrAdded/OnItemAdded event notification".
And, as seen in bug　917769, Gloda surely does do something upon "filter move at pop3 mail download", and as we already know well, "newly added mail by download/filter move or copy" is immediately/automatically shown at Thread Pane...


Matthew Jurgens(bug opener), no error\warning message or information message by Gloda in Error Console before you know "rebuild-index" by Virtual Folder open?

Comment 210

[WADA:World Anti-bad-Duping Agency]

FYI.
SetSummaryValid(true or false) is called at:
> http://mxr.mozilla.org/comm-central/search?string=SetSummaryValid%28&find=&findi=&filter=^[^\0]*%24&hitlimit=&tree=comm-central
GetSummaryValid(&valid) is called at:
> http://mxr.mozilla.org/comm-central/search?string=GetSummaryValid%28&find=&findi=&filter=^[^\0]*%24&hitlimit=&tree=comm-central
GetSummaryValid()/SetSummaryValid() is defined in nsMailDatabase.cpp
> http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMailDatabase.cpp#100
SetSummaryValid calls nsMsgDatabase::SetSummaryValid of nsMsgDBService/nsIMsgDBService
> http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#4075

Comment 211

[WADA:World Anti-bad-Duping Agency]

While testing new version of my addon with "filter move upon pop3 download" for this bug, I was aware of phenomenon of bug 762318(and bug 935934). See bug 935934 comment #4 for detail, please.

In MsgDB Event log by my addon, interesting phenomenon was observed.

(1) With "Filter before Junk Classification" (my ordinal setting)
    3 mails are downloaded and moved by filter.
    messageKey = previous key + 1. messageOffset = Inbox file size.
    No "EventType=DeleteOrMoveMsgCompleted" notification(Folder Event by Folder Listener)
    bug 762318(and bug 935934) occurs.
> DateTime=2014/03/19 06:51:40.190 GMT, EventType=DBOpened, MsgDB=mailbox://[e-mail]/Sent
> DateTime=2014/03/19 06:51:40.192 GMT, EventType=DBOpened, MsgDB=mailbox://[e-mail]/Sent
> DateTime=2014/03/19 06:51:40.193 GMT, EventType=DBOpened, MsgDB=mailbox://[e-mail]/Sent
> DateTime=2014/03/19 06:51:40.195 GMT, EventType=DBOpened, MsgDB=mailbox://[e-mail]/Sent
> DateTime=2014/03/19 06:51:40.196 GMT, EventType=DBOpened, MsgDB=mailbox://[e-mail]/Sent
> DateTime=2014/03/19 06:51:40.198 GMT, EventType=DBOpened, MsgDB=mailbox://[e-mail]/Sent
> DateTime=2014/03/19 06:51:40.200 GMT, EventType=MsgDBHdrAdded, messageKey=1233, messageOffset=1100, messageSize=595, messageKey>messageOffset=true, MsgDB=mailbox://[e-mail]/Sent
> DateTime=2014/03/19 06:51:40.245 GMT, EventType=MsgDBHdrAdded, messageKey=1234, messageOffset=1100, messageSize=595, messageKey>messageOffset=true, MsgDB=mailbox://[e-mail]/Sent
> DateTime=2014/03/19 06:51:40.286 GMT, EventType=MsgDBHdrAdded, messageKey=1235, messageOffset=1100, messageSize=599, messageKey>messageOffset=true, MsgDB=mailbox://[e-mail]/Sent

(2) With "Filter after Junk Classification"
    3 mails are downloaded and moved by filter.
    messageKey == messageOffset. messageOffset = Offset in move target folder. 
    "EventType=DeleteOrMoveMsgCompleted" is notified to Folder Listener.
    This is same as "manual filter run"(after the fact filter), manual copy/move of mail.
      i.e. "Batch type DoCopy" is used.
    bug 762318(and bug 935934) doesn't occur.
> DateTime=2014/03/19 06:57:14.558 GMT, EventType=MsgDB_was_Closed, MsgDB=mailbox://[e-mail]/VF-Sent, Info=Closed between 2014/03/19 06:56:14.557 GMT and 2014/03/19 06:57:14.557 GMT
> DateTime=2014/03/19 07:21:40.942 GMT, EventType=DBOpened, MsgDB=mailbox://[e-mail]/VF-Sent
> DateTime=2014/03/19 07:21:40.944 GMT, EventType=DBOpened, MsgDB=mailbox://[e-mail]/VF-Sent
> DateTime=2014/03/19 07:21:40.945 GMT, EventType=DBOpened, MsgDB=mailbox://[e-mail]/VF-Sent
> DateTime=2014/03/19 07:21:40.946 GMT, EventType=DBOpened, MsgDB=mailbox://[e-mail]/VF-Sent
> DateTime=2014/03/19 07:21:40.947 GMT, EventType=MsgDBHdrAdded, messageKey=7103, messageOffset=7103, messageSize=495, MsgDB=mailbox://[e-mail]/Sent
> DateTime=2014/03/19 07:21:40.964 GMT, EventType=MsgDBHdrAdded, messageKey=7598, messageOffset=7598, messageSize=595, MsgDB=mailbox://[e-mail]/Sent
> DateTime=2014/03/19 07:21:40.991 GMT, EventType=DeleteOrMoveMsgCompleted, FolderURI=mailbox://[e-mail]/Inbox


Matthew Jurgens(bug opener), do you see this bug with "Filter after Junk Classification"?
 note: 
 as noted in bug 762318, at "filter move target folder",
 "new icon" is perhaps not shown for "mail moved by filter upon pop3 download",
 because "move target folder" is opened once for "filter move after download to Inbox"
 then opened again by "mail viewing".

Comment 212

[Matthew Jurgens]

(In reply to WADA from comment #209)
> Matthew Jurgens(bug opener), no error\warning message or information message
> by Gloda in Error Console before you know "rebuild-index" by Virtual Folder
> open?
No entries in the error console

Comment 213

[Matthew Jurgens]

(In reply to WADA from comment #211)
> Matthew Jurgens(bug opener), do you see this bug with "Filter after Junk
> Classification"?
I think I do.
I have a very recent filter log entry like this:
Applied filter "eBay" to message from - at 1/01/1970 11:00:00 AM moved message id = to mailbox://mail.edcint.co.nz%3Amjurgens%40edcint.co.nz@gw/Bulk/eBay 

I have other filter log entries like these:
 Applied filter "eBay" to message from "Impression Real Estate" <[e-mail]> - HOT IN THE CITY at 4/03/2014 1:41:02 PM moved message id = [e-mail] to mailbox://mail.edcint.co.nz%3Amjurgens%40edcint.co.nz@gw/Bulk/eBay

Applied filter "eBay" to message from <[e-mail]> - Team Site at 26/02/2014 7:34:23 AM moved message id = [e-mail] to mailbox://mail.edcint.co.nz%3Amjurgens%40edcint.co.nz@gw/Bulk/eBay 

These are strange because the filter "eBay" was not really applied and these messages remained in my Inbox (as they should have).

Comment 214

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #213)
> Applied filter "eBay" to message from - at 1/01/1970 11:00:00 AM moved message id = to
> mailbox://mail.edcint.co.nz%3Amjurgens%40edcint.co.nz@gw/Bulk/eBay

It's bug 762318(and bug 935934). (your local time=GMT+11:00, according to mail from you.)

> These are strange because the filter "eBay" was not really applied (snip)

It is pretty natural in Tb.
  Because "filter log" is written by Tb before actually invoke "filter move operation",
  if "actual filter move" failed by unknown reasons,
  phenomenon of "Filter log lies" can occur.
  This "'message filter move' silently fails" and/or "Junk filter move' silently fails"
  is well known(but cause nor detail of phenomenon is still not known well).
  And, it's pretty old problem. 

> (snip) and these messages remained in my Inbox (as they should have).(snip)

When you saw the "these messages remained in my Inbox (as they should have)", was message mentioned in bug 931303 shown in your environent?
Or no message was shown, but "filter move" silently failed and kept message in Inbox without "moving mail to move target folder", even though "filter log" says "the message was moved to move target folder"?

By the way, this bug is pretty important, because isolation of next cases is needed.
  (a) true  "rebuild-index is mandatory"    : actual "outdated msf" condition
  (b) true  "rebuild-index is never needed" : ordinal, usual situation
  (c) false "rebuild-index is needed"       : this bug
  (d) false "rebuild-index is not needed"   : bug 261419 and bug 495760
To distinguish (c) from (a)/(b)/(d), we need to know about "a phenomenon is (a) or (b) or (c) or (d)".
Therefore, we have to learn about (a)/(b)/(d) well, before analysis of problems in (c) :-)

Comment 215

[Matthew Jurgens]

(In reply to WADA from comment #214)
> It is pretty natural in Tb.
>   And, it's pretty old problem. 
> When you saw the "these messages remained in my Inbox (as they should
> have)", was message mentioned in bug 931303 shown in your environent?
> Or no message was shown, but "filter move" silently failed and kept message
> in Inbox without "moving mail to move target folder", even though "filter
> log" says "the message was moved to move target folder"?

Let me make this clearer: These are pretty strange log entry since the filter would not match these messages. The ebay filter looks for ebay in the subject, to and body. These messages did not have that string. So the filter is reporting as matching and working but it fails to match correct and it fails to work correctly.

Comment 216

[WADA:World Anti-bad-Duping Agency]

(In reply to Matthew Jurgens from comment #215)
> Let me make this clearer: These are pretty strange log entry since the filter would not match these messages.
> The ebay filter looks for ebay in the subject, to and body. These messages did not have that string.
> So the filter is reporting as matching and working but it fails to match correct and it fails to work correctly.

Message filter log of "move to folder, with no Subject:/From:/Messsage-ID: information, with timestamp of 1970/01/01"?
If so, it's phenomenon of bug 935934 comment #4.
If you add "Set Junk Status to / Not Junk" action before "Move to folder" action, you can know mail's Subject/From/Message-ID/Date information by filter log for "Set Junk Status to" step, even when "Filter before Junk Classicication".
If log for "Set Junk Status to" step, or if log for "Move to folder" step with "Filter after Junk Classification", such "phantom move to folder log" is not observed.

If "Filter after Junk Classification", messageKey==messageOffset=="Offset of moved mail in move target folder" is passed to HdrAdded event listener.
If "Filter before Junk Classification", messageKey=previous key + 1, messageOffset=="Offset of moved mail in Inbox" is passed to HdrAdded event listener.
I believe messageOffset=="Offset of moved mail in Inbox" is relevant to that bug.
It may be problem like;
  "Truncate at Inbox file size before append mail data" is executed immediately in "filter move".
  Truncate is done at the messageOffset=="Offset of moved mail in Inbox".
  => when filter logger tries to read mail data for logging, truncate is already executed at Inbox.

Comment 217

[WADA:World Anti-bad-Duping Agency]

FYI.
See attached data to bug 935934 comment #7 ( attachment 8395180 [details] ) for difference between "Filter after Junk Classification" and "Filter before Junk Classification" which is visible by log data.

Interesting things: 

(1) When "Filter after Junk Classification", summaryValid=false is temporarily set while doing "download mails to Inbox" and "filter move to move target filter", for both "message in Inbox" and "message in move target folder".
This is perhaps for "power failure/system crash while downloading many mails to Inbox" and "power failure/system crash while doing filter move of many mails after download".
This "summaryValid=false" is not seen after end of download/filter move.

When is summaryValid=false changed to summaryValid=true?

(2) When "Filter before Junk Classification", (a) messageKey = Highest messageKey + 1 is used.
    And, (b) messageOffset=Offset in Inbox(==file size of Inbox before append mail data) is returned
    when HdrAdded event is notified to MsgDB Event Listener.
    This is not observed when "MsgDBHdr of moved mail in move target folder" is checked after
    download/filter-move ended. "messageOffset=Offset in move target folder" is always returned 
    for mail in the local move target folder.
I already knew (a) by showing "Order Received" column. This is new behavior from Tb 12. But I didn't know about (b) until I created MsgDB Event Listener code for this bug and checked MsgDBHdr.messageOffset value when HdrAdded event was notified to Event Listener.

When HdrAdded event is notified to event listeners?
Too early if "move to other folder by filter" occurs?

(3) When "move target folder" is not opened(MsgDB is not cached) upon mail download,
      If "Filter after Junk Classification", folder was always closed just after download/filter move.
      If "Filter before Junk Classification", "folder close just after end of download/filter move"
      was not seen in my test.

Why different?

Comment 218

[WADA:World Anti-bad-Duping Agency]

Attachment: Events on Sent folder for which summaryValid=false is observed

This is event log on Sent folder for which summaryValid=false is seen in log data(data sent from bug opener to me by mail).
"Events before summaryValid=false is observed" is following only.
  After restart of Tb, several "Open/Close"es
  HdrAdded. one mail only. Then close
  Immediate Open just after close
That all!

(1) 2014/03/19 09:23:26.359 Batch summaryValid check #1.
    Sent folder : summaryValid=true
(2) After it, open/close is repeated.
    No HdrAdded/HdrDeleted event.
(3) 2014-03-19 11:07:02.926000
    "add one mail to Sent" occurs.
    Sent folder and Unified Sent folder is opened.
    Because "HdrDeleted event log at Inbox" is not seen,
    this looks "save sent mail copy",
    but "save to Drafts" is not seen.
(4) After (3), "immediate close, then immediate re-open" is seen on
    Sent folder in MSGDB:5 log.
    This is pretty similar to #14 and #18.
(5) While Sent is still kept open,
    2014/03/19 11:07:31.182 batch summaryValid check #2.
    Sent folder : summaryValid=false
(6) After it,
    2014-03-19 11:08:08.676000 Sent is closed.
(7) 2014-03-19 11:12:27.208000 End of Tb log.

If summaryValid=false is kept by final folder close,
Rebuild-Index may be invoked after restart of Tb by Explicit Folder open, by Virtual Folder open etc.
If "filter move" accesses the Sent folder first after restart, "summaryValid=false status" may be lost.

Comment 219

[WADA:World Anti-bad-Duping Agency]

summaryValid status transition of Sent folder in log #19 sent to me.
(1) After restart, several open/close with no update of Sent folder content. Check summaryValid.
> == Batch summaryValid check #1 ==
> 2014/03/19 09:23:26.359 GMT
> msgDatabase.summaryValid = true, msgDatabase_is_Cached_when_checked = false,
> URI = mailbox://mail.edcint.co.nz%3Adefault.mailbox%40edcint.co.nz@gw/Sent
(2) One mail is added to Sent, then Sent is closed
> 2014-03-19 11:07:02.926000 UTC 
>   request f0ccc00 DoCopy - src  dest mailbox://mail.edcint.co.nz%3Adefault.mailbox%40edcint.co.nz@gw/Sent numItems 0 type=1
>   nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Matthew\[e-mail]\Sent.msf, FALSE, 83d7cf0, TRUE)
>   EventType=DBOpened, MsgDB=mailbox://mail.edcint.co.nz%3Adefault.mailbox%40edcint.co.nz@gw/Sent
>             summaryValid=true at this time, because event listeer checks summaryValid upon DPOpened event.
>   nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\smart mailboxes\Sent.msf, FALSE, c9e2410, TRUE)
>   EventType=DBOpened, MsgDB=mailbox://nobody@smart%20mailboxes/Sent
>             summaryValid=true at this time, because event listeer checks summaryValid upon DPOpened event.
>   EventType=MsgDBHdrAdded, messageKey=2126627, messageOffset=2126627, MsgDB=mailbox://mail.edcint.co.nz%3Adefault.mailbox%40edcint.co.nz@gw/Sent
>   NotifyCompletion - src  dest mailbox://mail.edcint.co.nz%3Adefault.mailbox%40edcint.co.nz@gw/Sent
>   request f0ccc00 Clearing OK request - src  dest mailbox://mail.edcint.co.nz%3Adefault.mailbox%40edcint.co.nz@gw/Sent numItems 0 type=1
> 2014-03-19 11:07:03.614000 UTC
>   closing database    S:\data\Mozilla Thunderbird\Matthew\[e-mail]\Sent.msf
(3) After close, Sent is immediately opened again. By Gloda?
> 2014-03-19 11:07:03.946000 UTC
>   EventType=DBOpened, MsgDB=mailbox://mail.edcint.co.nz%3Adefault.mailbox%40edcint.co.nz@gw/Sent
>             summaryValid=true at this time, because event listeer checks summaryValid upon DPOpened event.
> 2014-03-19 11:07:03.958000 UTC
>   nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Matthew\[e-mail]\Sent.msf, FALSE, 7ee9970, TRUE)
(4) Sent folder is kept open. summaryValid is checked, and summaryValid=false is detected.
> == Batch summaryValid check #2 ==
> 2014/03/19 11:07:31.182    GMT
>  [msgDatabase.summaryValid] = false  
>  [Timestamp] = 2014/03/19 11:07:31.182 GMT 
>  [FolderURI] = mailbox://mail.edcint.co.nz%3Adefault.mailbox%40edcint.co.nz@gw/Sent

(Question-1)
Event on Sent folder is one time of "one sent mail copy is saved in Sent" only. That's all!
There is no "filter move" in this case. "DoCopy without src folder" only is involved.
Why summaryValid=false is set merely by "one sent mail copy"?

I can't imagine culprit other than "immediate open of Sent by someone just after close by 'save to Sent'" in this case.
Note: "immediate re-open just after close" is seen on Spam in log #14, Trash in log #18 too.

(Question-2)
My MsgDB Event Listener checks summaryValid value upon each DBOpened event, and write log for summaryValid=false if summaryValid=false is detected.
There is no log for "summaryValid=false upon DBOpened event".
i.e. summaryValid is set by someone after last DBOpened event notification at 11:07:03.946000.
Funny phenomenon is;
  DBOpened event is notified at 11:07:03.946000, before nsMsgDatabase::Open is logged at 11:07:03.958000.
When nsMsgDatabase::Open log is written, "Open Sent.msf file and full read of Sent.msf file" is always started just before nsMsgDatabase::Open log or after nsMsgDatabase::Open log.
And, DBOpened event should be notified after end of "full reead of Sent.msf" and end of "re-construction of MsgDB data in memory from .msf file content".
DBOpened event is usually notified after 20msec from nsMsgDatabase::Open log, if Sent folder in this log.
Why DBOpened event is notified before nsMsgDatabase::Open log?


Note-1:
msgDatabase_is_Cached_when_checked is :
      var cachedDB=MsgDBService.cachedDBForFolder(msgFolder);
      if(cachedDB) msgDatabase_is_Cached_when_checked = true;
      else         msgDatabase_is_Cached_when_checked = false;
Because property of msgFolder.msgDatabase is accessed at same time for msgFolder.msgDatabase.summarValid value check, if msgDatabase_is_Cached_when_checked=false status, msgDatabse is opened and cached by this listing.
Note-2:
MsgDB Event Listener checks msgDatabase,summaryValid value when DBOpened event is notified.
Following phenomenon is observed by it some times.
  1. DBOpened event #1 is notified. msgDatabase,summaryValid is checked and true is returned.
  2. DBOpened event #2 is somehow notified. msgDatabase,summaryValid is checked and true is returned.
  3. It's not so frequently but "more than 2 times of DBOpened event" is also observed.
I think this is nexr phenomenon:
  When DBOpened event #1 is notified, MsgDB is normally re-constructed in memory.
  But "DB is Cached" state is not set completely yet.
  If msgDatabase.summaryValid is accessed by DBOpened #1 event listner before "DB is Cached" state
  is set, "MsgDB open logic" is partially executed instead of "quickly return cached DB"
  upon "internal open by msgDatabase.summaryValid access", then DBOpened event #2 is notified.

Comment 220

[WADA:World Anti-bad-Duping Agency]

Attachment: MSGDB:5/MsgPurge:5 log, MsgDB Event log, for "filter move when outdated msf condition exists"

This is log for:
(0) "outdated msf condition" exists in Test/DDD
(1) Download 3 mails.
(1-1) Due to "outdated msf", "before Classification" filter doesn't move mail
(1-2) So, mails are held in Inbox
(1-3) "after Classification" filter moves mails to Test/DDD,
       without creating MsgDHdr for moved mail,
       and delete mails from Inbox.
(2) Open Text/DDD by Virtual Folder open.
    => Rebuild-Index is executed.

What is found by this test.
  - We can't know "outdated msf condition etc. exists or not" by MSGDB:5 log.
  - We can know only following by MSGDB:5 log:
    - When "MsgDatabase open request log" is written.
    - When "closing database" log is written.
    - When "error opening db" log is written.
    - When "nn open DBs" log is written,
      what DB is opened, and number of referred MsgDBHdr at this time.
  - Because "MsgDatabase open request log" is written for BackupDatabase,
    and because BackupDataBase is used by rebuild-index,
    we can assume that "MsgDatabase open request log for BackupDatabase"
    is an evidence of "rebuild-index was executed".
  - If MsgCopyService:5 log is added, we can estiate "when filter move was
    invoked".
  - If MsgDB Event Listeer log(DBOpened, HdrAdded/HdrDeleted etc.) is added,
    "understanding MSGDB:5 log and MsgCopyServce:5 log" is easier
    than "MSGDB:5 log and MsgCopyServce:5 log only".

I can find no difference between "log #14, #18, #19 by bug opener" and "this log by me".
I can say next only.
- If "MsgDatabase open request log" is written for BacupDatabase 
  by MSGDB:5 logging,
  it's perhaps evidence of "rebuild-index was surely invoked by someone".
- If "opener of the MsgDB"(who invoked MsgDatabase open request log) is
  "who invokes rebuild-index when he detects outdated msf condition"
  (eg. Explicit Folder Open, Virtual Folder),
  and if rebuild-index was executed by the "opener of MsgDB",
  "outdated msf condition" perhaps surely existed when MsgDB was opened.

Comment 221

[WADA:World Anti-bad-Duping Agency]

FYI.

When "outdated msf condition" exists in BBB.msf, and when BBB folder is not opened,
if msgDatabase.summayValid is accessed by JavaScript code, following occurs.
1. nsMsgDatabase::Open log is written by MSGDB:5 logging.
2. "closing database" log is immediately written by by MSGDB:5 logging.
3. "Exception with 0x80550005" is returned to JavaScript code.
4. Because actual "MsgDatabase open" is not executed due to "outdated msf condition",
   DBOpened event is not notified to MsgDB Event Listener.

It seems that "immediate close log, just after open log with no error" is an indicator of "outdated msf condition" or something bad.

> (0) "Outdated msf condition" already exists in Test/BBB
>      Note: Batch msgDatabase.sumarryValid check routine now writes
>              EventType=summaryValid_Error_was_found_by_batch log.
>            Info part is Exception_Event.message data.
> 
> (1) Batch msgDatabase.sumariValid check #1
> 
> 2014-03-24 08:45:05.782000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\ ... \Mail\Local Folders\Test.sbd\BBB.msf, FALSE, 6664110, TRUE)
> 2014-03-24 08:45:05.798000 UTC - 0[e0f140]: closing database    C:\ ... \Mail\Local Folders\Test.sbd\BBB.msf
> 2014-03-24 08:45:05.798000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\ ... \Mail\Local Folders\Test.sbd\BBB.msf, FALSE, 6664110, TRUE)
> 2014-03-24 08:45:05.798000 UTC - 0[e0f140]: closing database    C:\ ... \Mail\Local Folders\Test.sbd\BBB.msf
> 2014-03-24 08:45:05.818000 UTC - 0[FFFFFF]: EventType=summaryValid_Error_was_found_by_batch, MsgDB=mailbox://nobody@Local%20Folders/Test/BBB, Info=Component returned failure code: 0x80550005 [nsIMsgFolder.msgDatabase]
> 
> (2) Batch msgDatabase.sumariValid check #2
> 
> 2014-03-24 08:46:01.235000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\ ... \Mail\Local Folders\Test.sbd\BBB.msf, FALSE, 6664830, TRUE)
> 2014-03-24 08:46:01.235000 UTC - 0[e0f140]: closing database    C:\ ... \Mail\Local Folders\Test.sbd\BBB.msf
> 2014-03-24 08:46:01.235000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\ ... \Mail\Local Folders\Test.sbd\BBB.msf, FALSE, 6664830, TRUE)
> 2014-03-24 08:46:01.235000 UTC - 0[e0f140]: closing database    C:\ ... \Mail\Local Folders\Test.sbd\BBB.msf
> 2014-03-24 08:46:01.252000 UTC - 0[FFFFFF]: EventType=summaryValid_Error_was_found_by_batch, MsgDB=mailbox://nobody@Local%20Folders/Test/BBB, Info=Component returned failure code: 0x80550005 [nsIMsgFolder.msgDatabase]
> 
> (3) Batch msgDatabase.sumariValid check #2
> 
> 2014-03-24 08:46:59.001000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\ ... \Mail\Local Folders\Test.sbd\BBB.msf, FALSE, 6664cf0, TRUE)
> 2014-03-24 08:46:59.017000 UTC - 0[e0f140]: closing database    C:\ ... \Mail\Local Folders\Test.sbd\BBB.msf
> 2014-03-24 08:46:59.017000 UTC - 0[e0f140]: nsMsgDatabase::Open(C:\ ... \Mail\Local Folders\Test.sbd\BBB.msf, FALSE, 6664cf0, TRUE)
> 2014-03-24 08:46:59.017000 UTC - 0[e0f140]: closing database    C:\ ... \Mail\Local Folders\Test.sbd\BBB.msf
> 2014-03-24 08:46:59.024000 UTC - 0[FFFFFF]: EventType=summaryValid_Error_was_found_by_batch, MsgDB=mailbox://nobody@Local%20Folders/Test/BBB, Info=Component returned failure code: 0x80550005 [nsIMsgFolder.msgDatabase]

What a useful MSGDB:5 log!!!
I thought "something indicates error" is written by MSGDB:5 logging if "outdated msf condition" exists...
What is purpose of current "error opening db 0x8055000X" logging?

Comment 222

[WADA:World Anti-bad-Duping Agency]

FYI.

As I wrote in bug 931303 comment #58, following can be used as "automatic manual filter run feature for bug 931303".
  Rule set rule-N : consists of 3 rules.
     rule-N-0 : before Classification, if condition-N, do actions except Move to Folder
     rule-N-1 : before Classification, if condition-N, Move to Folder-N
     rule-N-2 : after  Classification, if condition-N, Move to Folder-N
Phenomenon of bug 931303 can be used as "outdated msf condition detector in filter move" for this bug.
  If "outdated msf condition" exists in Folder-N upon download/filter move,
  error message is shown at rule-N-1(before Classification) application step.
This is pretty useful "outdated msf condition detector" for you.
If both "before Classification, Move to Folder-N" and "after Classification, Move to Folder-N" are defined with same condition, there is no need to disable "check new messages upon startup" for checking/testing.

Comment 223

[WADA:World Anti-bad-Duping Agency]

I can now understand following MSGDB:5 log for Spam in 14.Sent.Spam.TB.log.
> Someone like my addon tries to open Spam or triedsto access msgDatabase.xxx, 
> and the someone does do nothing when RC="outdated msf condition" is returned.
> 2013-10-11 22:22:46.463000 UTC  nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf, FALSE, 727c410, TRUE)
> 2013-10-11 22:22:46.588000 UTC  closing database    S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf
> 2013-10-11 22:22:46.603000 UTC  nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf, FALSE, 727c410, TRUE)
> 2013-10-11 22:22:46.697000 UTC  closing database    S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf
> 
> Virtual Folder opens Spam folder in order to show Spam folder content
> 
> 2013-10-11 22:22:56.463000 UTC  nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf, FALSE, 8fe68e0, TRUE)
> 2013-10-11 22:22:56.572000 UTC  closing database    S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf
> 2013-10-11 22:22:56.681000 UTC  nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf, FALSE, 8fe68e0, TRUE)
> 
> Virtual Folder correctly processes RC="outdated msf condition" for "Folder Open" call.
> Then Virtual Folder calls RebuilIndex() properly.
> 
> 2013-10-11 22:22:56.759000 UTC  nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\Spam.msf, FALSE, 8fe6a10, TRUE)
> 2013-10-11 22:22:56.759000 UTC  error opening db 80550006
>
> 2013-10-11 22:23:03.838000 UTC  closing database    S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\Local Folders\Spam.msf

Problem is;
  Following is pretty easily known by available data;
    When "outdated msf condition" exists,
      Someone calls RebuildIndex()                         <= Virtual Folder
      Someone does do nothing                              <= My addon
      Someone silently raises "msf data corruption level"  <= Filter Move(after  Classification)
      Someone shows pretty misleading error message        <= Filter Move(before Classification)
  But no one can know "when/how/by whom the outdated msf condition was produced" by available data...

Comment 224

[WADA:World Anti-bad-Duping Agency]

FYI.
"Compact Folders" was a someonw who sets non-temporary summaryValid=false.
(0) FolderX : 3 mails => delete 1 mail, expungedBytes=1000, sideOnDisk=3000
    append a mail data when folder is not opened
                       => status is :   expungedBytes=1000, sideOnDisk=4000
                          but "outdated msf condition" exists
(1) Compact Folders
    Folder Event = CompactCompleted is notified to Folder Listener.
    MsgDatabase is opened, and is kept open because no one close MsgDB
(2) Check summaryValid etc. by JavaScript
    summaryValid=false, expungedBytes=0, sideOnDisk=4000
    content of FolderX file is not changed
(3) Wait for MsgDatabase close
(4) Check summaryValid etc. by JavaScript after MsgDatabase is closed
    ExceptionEvent[message] = Component returned failure code: 0x80550005 [nsIMsgFolder.msgDatabase]
    expungedBytes=0, sideOnDisk=4000
    Because of "0x80550005", MsgDB is not opened.
(5) If FolderX is opened by Virtual Folder, rebuild-index is execued.

Above is perhaps behavior after patch for Bug 710056 was landed on Tb 18.
(note: Bug 750569 was closed as dup of Bug 710056)
> Bug 710056 - Earlybird doesn't remember columns change in the message list after restart
>  (only after restart of Tb which is automatically initiated when an update is applied by Help/About/Apply Update)
> Bug 750569 - Compact Folders window appearing very frequently starting in TB12

Comment 225

[WADA:World Anti-bad-Duping Agency]

Attachment: Log #20, Bulk/eBay, MSGDB:5,MsgCopyService:5 log, MsGDB Event Listener log

Log #20. Rebuild-Index occurred Bulk/eBay.
  open/close log by MSGDB:5   : log for Bulk.sbd\eBay.msf only
  log by MsgCopyService:5     : all log is printed
  log by MsgDB Event Listener : all log is printed

(1) Update of Bulk/eBay is following only.
 (a) 1 HdrAdded event only.
 (b) messageKey>messageOffset, no log for Inbox
     => filter move, before Classification
 (c) If filter move with before Classification,
     close is immediately occurs after HdrAdded event.
     However, "closing database" is logged after 2 minutes.
> 06:58:22.404000 UTC - 0[170f140]: nsMsgDatabase::Open(S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Bulk.sbd\eBay.msf, FALSE, be31090, TRUE)
> 06:58:22.452000 UTC - 0[FFFFFF]: EventType=DBOpened, MsgDB=mailbox://mail.edcint.co.nz%3Amjurgens%40edcint.co.nz@gw/Bulk/eBay
> 06:58:22.455000 UTC - 0[FFFFFF]: EventType=MsgDBHdrAdded, messageKey=22642680, messageOffset=12341589, messageSize=68597, messageKey>messageOffset=true, MsgDB=mailbox://mail.edcint.co.nz%3Amjurgens%40edcint.co.nz@gw/Bulk/eBay
> 07:00:39.857000 UTC - 0[170f140]: closing database    S:\data\Mozilla Thunderbird\Profiles\Matthew\Mail\edcint.co.nz\Bulk.sbd\eBay.msf
(2) After HdrAdded log and "closing database" log, all open/close is "open, then immediately close", and is repeated many times.
(3) Finally, Bulk/eBay is opened(perhaps by Virtual Folder), and Rebuild-Index is executed.
summaryValid=false and EventType=AnnouncerGoingAway is result of this Rebuild-Index.

If "outdated msf condition" already exists before filter move, HdrAdded event is not notified because creation of MsgDHdr fails. But HdrAdded event is logged.
=> Before filter move, no problem.

After "close Bulk/eBay after filter move", all open/close = "open, then immediately close".
=> "outdated msf condition" was generated during Bulk/eBay open or upon close.

If "filter move with before Classification", folder is usually immediately closed after filter move. However, "closing database" log is written after 2 minutes.

Above is similar to(rather same as) Sent case in log #14.

Conclusion.
- "Close just after filter move with before Classification" request was
  interfered by someone else, then Bulk/eBay was kept open.
  After a while, Bulk/eBay was closed by idle time threshold.
- If "Close just after filter move with before Classification" is interfered,
  state which produces "outdated msf condition" is generated.
- Because this bug is seen on "filter move with after Classification" folder,
  "after Classification" or "before Classification" is irrelevant to problem.
- "Close log just after filter move" is seen in other than Sent in #14 case.
  However, "re-open by other one just after close by filter move" is seen in
  the all other cases. And, after it, "access before Rebuild-Index" is
  "open, then immediately close". This is characteristics of "access when
  outdated msf condiion" exists.
  I think "closing database log is written by close request from filter move,
  or not" depends on timing.
             open=>OK         open=>Bad                 open=>Bad      open=>OK
              |                |                         |              |
              V                V                         V              V
  filter move -> close request -> "closing database" log -> end of close
                                   
Major suspects of "interfere of Close by filter move" :
- MsgDB Event Listener of my Addon.
   when DBopened/HdrAdded event is notified, my addon accesses msgDatabase.xxx.
   because access to msgDatabase property generates "Folder Open"
   if folder is closed(not cached), this may produce "open request while close
   is being processed".
     Evidence:
       if my addon checks msgFolder.msgDatabase.xxx when MsdDB is not
       cached, database open log is written by MSGDB:5 logging.
- Gloda
    Gloda tracks HdrAdded event, and starts indexing when MsgDBHdr is added.
    Evidence: Bug 917769
- Virtual Folder
    Virtual Folder tracks HdrAdded/HdrDeleted event,
    and immediately reflect it to Folder Pane/Thread pane.
    Evidence:
       Virtual Folder FolderV : search target = FolderA, FolderB
       Filter : Match all, Move to FolderB
       FolderA : 5000 unread mails
       FolderB : 0 mails
     Run Filter on FolderA
       Unread count of FolderB and FolderV is continuously increased
       Thread Pane of FolderB and FolderV is continuously updated.
       Unread count of FolderA is not updated while moving.
       Unread count is updated at end of bulk move.
       Unread count of FolderV is doubled(10000) by filter move,
       and is reduced to half(5000) after end of bulk move.
       HdrAdded/HdrDeleted events:
         At FolderB, 5000 HsrAdded is logged,
         then, at FolderA, HdrDeleted is logged.

Comment 226

[WADA:World Anti-bad-Duping Agency]

Difference between "Filter before Junk Classification" and "Filter after Junk Classification".
"Filter after Junk Classification"
  start of download
  download and add mail#1 to Inbox -> HdrAdded event for mail#1 at Inbox
  download and add mail#2 to Inbox -> HdrAdded event for mail#2 at Inbox
  Filter move by "Bulk Move" starts
    open Target
    copy   mail#1 from Inbox to Target -> HdrAdded event for mail#1 at Target
    copy   mail#2 from Inbox to Target -> HdrAdded event for mail#2 at Target
    delete mail#1 from Inbox -> HdrDeleted event for mail#1 at Inbox
    delete mail#2 from Inbox -> HdrDeleted event for mail#2 at Inbox
    close Target
"Filter before Junk Classification"
  start of download
  download mail #1, write data of mail#1 to Inbox
  open Traget
  create MsgDBHdr#1 for mail#1 in Target -> HdrAdded event for mail#1 at Target
  append data of mail#1 to Target
  truncate Inbox at original file size
  download mail #2, write data of mail#2 to Inbox
  (if move to different Target, is folder closed? or kept open?)
  create MsgDBHdr#2 for mail#2 in Target -> HdrAdded event for mail#2 at Target
  append data of mail#2 to Target
  truncate Inbox at original file size
  close Target

I think "closing database log is written, or not" is;

When multiple mails are downloaded and moved by filter, if "after Classification", filter move is executed after download of all mails, so time between open and close is short. Then, closing log is written before open is requested by process which was kicked by first HdrAdded event.
When multiple mails are downloaded and moved by filter, if "before Classification", filter move is executed for each mail after download of each mail, so "time between open and close" involves "download time". Then it's longer than "after Classifiction". Thus, before "closing database" log is written, open is requested by process which was kicked by first HdrAdded event.

Comment 227

[WADA:World Anti-bad-Duping Agency]

"closing database" log was written in Destructor of nsMsgDatabase C++ object.
> http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#1129
> 1129 nsMsgDatabase::~nsMsgDatabase()
> 1144   PR_LOG(DBLog, PR_LOG_ALWAYS, ("closing database    %s\n",
> 1145     (const char*)m_dbName.get()));
> Destructor
> https://publib.boulder.ibm.com/infocenter/lnxpcomp/v8v101/index.jsp?topic=%2Fcom.ibm.xlcpp8l.doc%2Flanguage%2Fref%2Fcplr380.htm

Destructor is invoked by delete MsgDB object. MsgDB object is perhaps created by nsMsgDatabse MsgDB = nsMsgDatabase(...) like call.
Difference between "closing database is written" and "closing database log is not written".

  After end of DoCopy, force DB close                 Other process
  delete MsgDB                                          after HdrAdded event notification,
    nsMsgDatabase::~nsMsgDatabase()                     try to access Msg of messageKey=X
      steps before "closing database" log is executed
                                                     <= Open request invoked by access to message
                                                        before "close database" log.
                                                        => "close database" log is not written.
                                                            MsgDB is kept open, without close.
      "closing database" is logged
                                                     <= Open request invoked by access to message
                                                        after "close database" log.
                                                        => "immediate open" is logged
      steps after "closing database" log is executed
    nsMsgDatabase::~nsMsgDatabase() ends
  C++ object is deleted.
  But still alive until garbage collector runs

If this kind of problem, it's a seriarization problem between "Open" and "Close", or, between "create object" and "destroy object".
Mutex lock like mechanism is needed?
Or "non-mandatory forced DB close after DoCopy" should be avoided before such serialization?

Comment 228

[WADA:World Anti-bad-Duping Agency]

Matthew Jurgens, you and me may have seen the moment baby of "outdated msf condition" is born :-)

Comment 229

[Wayne Mery (:wsmwk)]

(In reply to WADA from comment #227)
>...
> If this kind of problem, it's a seriarization problem between "Open" and 
> "Close", or, between "create object" and "destroy object".
> Mutex lock like mechanism is needed?
> Or "non-mandatory forced DB close after DoCopy" should be avoided before such 
> serialization?

(In reply to WADA from comment #228)
> Matthew Jurgens, you and me may have seen the moment baby of "outdated msf
> condition" is born :-)

reminder - we think this is a TB 3.0 regression, circa 2009.  So is the regressing code one of these? 
http://hg.mozilla.org/comm-central/log/66b59dea21b7/mailnews/db/msgdb/src/nsMsgDatabase.cpp

and (a shot in the dark) does this help point us in the direction of what is causing the often bad database memory reporting numbers?

Comment 230

[Wayne Mery (:wsmwk)]

(In reply to WADA from comment #177)
> "Outdated msf condition" is detected by
> nsMsgBrkMBoxStore::IsSummaryFileValid.
>...
>   For example,
>   while MsgDB caching process for msgDatabase open is in progress by task#1
> and state is
>   still cache-requested-but-not-completed-yet, msgDatabase open is requested
> by task#2.
> If this kind of problem, "problem started to occur from Tb 3" can be
> explained, because "msgDatabase close by idle time threshold"(.msf file
> close while Tb is running) was introduced by Tb 3(till Tb 2, .msf file was
> closed only when shutdown of Tb), and because ".msf file open/read entire
> data from .msf file for msgDatabase open" and "writing back .msf file" takes
> far longer than local HDD file when network file is used.

wada, are you able to identify the TB3 checkin/bug number that you are thinking of for close on idle?   Because I'm not remembering that this happened in TB3. I checked bugzilla [1] (note bug 470221 was closing for autosync only).  However we do have closing db on idle in TB15 with bug 723248.

[1] TB3 checkins https://bugzilla.mozilla.org/buglist.cgi?f1=short_desc&list_id=10777168&short_desc=db%20folder&bug_severity=major&bug_severity=normal&bug_severity=enhancement&o1=nowordssubstr&resolution=FIXED&classification=Client%20Software&classification=Components&chfieldto=2009-10-01&chfield=resolution&query_format=advanced&chfieldfrom=2008-07-07&f2=OP&chfieldvalue=fixed&longdesc=idle%20clos&short_desc_type=anywordssubstr&component=Backend&component=Database&target_milestone=Thunderbird%203.0a3&target_milestone=Thunderbird%203.0b1&target_milestone=Thunderbird%203.0b2&target_milestone=Thunderbird%203.0b3&target_milestone=Thunderbird%203.0b4&target_milestone=Thunderbird%203.0rc1&target_milestone=Thunderbird%203&target_milestone=---&target_milestone=Thunderbird%203.0a3&target_milestone=Thunderbird%203.0b1&target_milestone=Thunderbird%203.0b2&target_milestone=Thunderbird%203.0b3&target_milestone=Thunderbird%203.0b4&target_milestone=Thunderbird%203.0rc1&target_milestone=Thunderbird%203&target_milestone=---&longdesc_type=anywordssubstr&product=MailNews%20Core&product=Thunderbird

Comment 231

[Wayne Mery (:wsmwk)]

(In reply to WADA from comment #70)
> ...
> It may be interfere between Tb's tasks. For example,
>   POP3 account#1 downloads new mails, and filter moves mails to \Local
> Folders\Mbox.
>   At same time, POP3 account#2 downloads new mails, and filter moves mails
> to same Mbox.

> If Mbox is on local HDD, each "move mails to Mbox" is not so slow. However,
> if network drive, each "move mails to Mbox" takes pretty long, because one
> write request is issued by Tb for each mail data line(known bug).

If we have such cases (not saying that's the case here, yet) then bug 558528 will help.

Comment 232

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #231)

"one write request per each mail data line" is Bug 769346.
Bug 769346 is not "buffer size" problem.  Bug 769346 is problem of "buffering is not used".

Comment 233

[Wayne Mery (:wsmwk)]

Matthew, can you test the builds from these locations and tell us whether they reproduce the problem or not?
(backup your profile before testing)

(bug 511609 landed 2009-08-21)
https://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2009/08/2009-08-21-11-comm-central-trunk/
https://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2009/08/2009-08-23-03-comm-central-trunk/

Comment 234

[Matthew Jurgens]

I am very confident that it occurs on thunderbird-3.1a1pre.en-US.win32.installer.2009-08-23-03.exe.

I need a little longer on thunderbird-3.1a1pre.en-US.win32.installer.2009-08-21-11.exe.

Comment 235

[WADA:World Anti-bad-Duping Agency]

FYI.
According to bug 624806 comment #15, which was posted by Matthew Jurgens on 2012-01-16, which was answer to question of bug 624806 comment #7 from Wayne, bug 624806 started to occur after upgrade to Tb 3.0, instead of upgrade from Tb 3.0 to Tb 3.1.
Copy of bug 624806 comment #15 from Matthew Jurgens on 2012-01-16.
> I got the problem as soon as going to 3.0 and it is still with me at v9
I believe this bug(bug 905576) is for absolutely same problem as bug 624806. i.e. This bug is excellent version of bug 624806.
Note: IIUC, "Close MsgDB by threshold of inactivity period" is new feature from Tb 3.0, instead of from Tb 3.1.

Comment 236

[Matthew Jurgens]

The original getsatisfaction.com page that I wrote that contained a lot of detail about this bug is no longer available. I have put up an old version from 2012 here:
http://www.edcint.co.nz/tmp/TB3%20repeatedly%20rebuilding%20summary%20files%20for%20many%20folders%20when%20thunderbird%20profile%20is%20on%20network%20drive.%20No%20problem%20in%20v2.htm

By the time I posted that original page I had been having problems for a while and I included a bit of detail. On that page I talked about how this problem happened when moving from 2.x to 3.x Thunderbird.

Whilst I am having some trouble reproducing the problem on thunderbird-3.1a1pre.en-US.win32.installer.2009-08-21-11.exe this might be because of the way I am trying to reproduce the problem. That way of reproducing the problem (disabling retrieval of mail from my main email accounts, setting up a test account with filter rules to move mail around on arrival) did eventually reproduce the problem on thunderbird-3.1a1pre.en-US.win32.installer.2009-08-23-03.exe. I will keep trying.

Comment 237

[Matthew Jurgens]

I have just managed to reproduce the same problem with thunderbird-3.1a1pre.en-US.win32.installer.2009-08-23-03.exe.

So both builds from comment #233 have the same problem as described by this bug report

Comment 238

[Matthew Jurgens]

Also, this bug has been reproduced with Windows 7 (SMB2).

Comment 239

[WADA:World Anti-bad-Duping Agency]

Setting dependency to bug 1135310.
> Bug 1135310 - closing idle folder databases potentially broken

Comment 240

[Matthew Jurgens]

I have performed multiple upgrades over the past few months. These have had a positive impact on the issue but not removed it entirely.

This bug was initially reported with the client running Thunderbird 3.x on Windows XP (SMB1) on an Intel Core 2 Duo E7500 and the server running Samba (v3.6) on Fedora 17 using an Intel Core 2 Quad E6600.
Now the systems have been upgraded to Thunderbird 45 on Windows 10 (SMB2+) on an Intel i7-6700 and the server running Samba (v4.3) on Fedora 23 using an Intel i5-6500.

These are major upgrades both in software and hardware and has resulted in a major CPU performance upgrade. The bug is now much harder to reproduce and seen less often (versus every day). In my mind this increases the likelihood that the bug is related to the timing window, which of course now, has been significantly reduced.

Although this bug will probably never be fixed, since Thunderbird is a lot more usable I am happy to remain with Thunderbird at this point.

Comment 242

[Matthew Jurgens]

I switched to "file per message" as soon as it was made available in the beta versions. That resolved it for me.

Comment 243

[ISHIKAWA, Chiaki]

This now sounds to me that the bug is caused by a short read due to SAMBA or CIFS server sometimes returning shorter than requested data.
The chance of this happening is decreased by the lesser load on the server (which means smaller amount of I/O for re-indexing, e.g. the use of maildir should help, I think.)

But unless someone can throttle the output length on the server side so that the write to the client is capped at a smaller size, say, 1K or lesser, and repeat the test, it is hard to say what is going on.