855751 - TB forgetting folder view and display prefs after running out of file handles on Mac. (also on linux with similar config) "The file XXXX could not be opened. ..."

Status: UNCONFIRMED

(MailNews Core :: Backend, defect)

Description

[Jim Moe]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:19.0) Gecko/20100101 Firefox/19.0
Build ID: 2013030600

Steps to reproduce:

Used TB normally.

The only exists on MacOS. Other OSes do not have this issue.

I posted this issue on the Mozilla/TB user forum. No responses. Here is the text:

iMac MacOS 10.8.2
TB v17.0.3

Recently a number of add problems have occurred with TB. Showing
mostly at startup, the message (paraphrased)

"The file <insert name here> could not be opened. It may held by another
program, marked as read-only, or not exist."

This has happened with various address book files <abook-NN.mab> and,
today, <imap-NN.msf> (NN is some number, it varies). Restarting TB
usually clears the problem for those particular files but sometimes
other ones cause a fault.

Other oddities I suspect are related to this are:
- Message list settings were not saved when quitting
- Reloading an account's folder structure at startup every time
- The spell checker highlighting every single word in a new message

  I zipped the profile, erased the profile directory, and restored it
from the ZIP archive. Some of the problems disappeared. Given the random
and flaky nature of the file permission problem, I expect to see them
return.



Actual results:

I repeatedly must Repair Folders, set the View::Sort By options (date descending, threaded), and change the Folder columns to my preferences.

All of these settings randomly disappear.


Expected results:

I should only need to do the above once.

Comment 1

[Wayne Mery (:wsmwk)]

What happens with Thunderbird 17.0.5 started in safe mode?
https://support.mozillamessaging.com/kb/safe-mode

Comment 2

[Jim Moe]

Starting Safe mode lead to a solution. The problem appears to be an addon: Provider for Google Calendar. After disabling that addon, the other odd problems have ceased, including the bit about access being blocked to an Address Book file. (It is not the least bit clear how the addon affected the filesystem, but there it is.)

Please give this a week (until 27-May) before declaring it resolved.

Comment 3

[Jim Moe]

On a further note regarding that addon... It functions without a problem in the openSUSE/linux version. It seems to have an adverse effect only in MacOS. (I cannot say about Windows.)

Comment 4

[Jim Moe]

No, that was not the problem. The View preferences and message window settings are still simply dumped and reset to the default values. This is an issue only on the MacOS port. The linux and Windows ports do not display the problem and have the same set of addons.

Comment 5

[Jim Moe]

I did resolve the Speller issue: I added a dictionary addon for English. Since then, at least the erratic speller problem has not returned.

Comment 6

[Phoenix]

Resolved per whiteboard and Comment 5

Comment 7

[Jim Moe]

No, it is not resolved. Only the speller aspect has a workaround. (Why did I need to add an add-on dictionary? What is wrong with the built-in?)

The other, more annoying issues remain unresolved.

Comment 8

[Jim Moe]

This issue appears to be related to this error message that shows in the Error Console:

Timestamp: 6/7/13 2:17:49 PM
Error: [Exception... "Component returned failure code: 0x80004003 (NS_ERROR_INVALID_POINTER) [nsIMsgFolder.getStringProperty]" nsresult: "0x80004003 (NS_ERROR_INVALID_POINTER)"  location: "JS frame :: chrome://messenger/content/folderPane.js :: getSmartFolderName :: line 2436"  data: no]
Source File: chrome://messenger/content/folderPane.js
Line: 2438


It shows dozens of times, grouped by regular intervals. It occurs at the same time that the folder appearance sometimes spontaneously reverts to default values.

Comment 9

[Wayne Mery (:wsmwk)]

Jim, did your solution in bug 495911 also solve the problem of this bug?

Comment 10

[Jim Moe]

I do not know yet. The set up profile decays with time, on the order of days. I went through all of the accounts last night, setting the display and view to what I prefer. Now I must wait several days before I can decide if there is any improvement.

I really, really, really wish there were a way to change the default settings to my own preferences so that when anything is reset to default, it is what I want.

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to Jim Moe from comment #7)
> (Why did I need to add an add-on dictionary? What is wrong with the built-in?)

The addon apparently loads the dictionary early in the startup process, prior to your Mac instance running out of file handles.  


(In reply to Jim Moe from comment #0)
> Recently a number of add problems have occurred with TB. 
> Showing mostly at startup, 

You mention Provider for Google Calendar. What other addons are installed, including disabled addons?


> the message (paraphrased)
> "The file <insert name here> could not be opened. It may held by another
> program, marked as read-only, or not exist."

Next time you see this, please cite the precise wording

Comment 12

[Jim Moe]

"Unable to load address book file abook.mab. It may be read-only, or locked by another application. Please try again later."

The filename <abook.mab> may also be <abook-3.mab> or <abook-10.mab>.

Is there any external way to increase the number of file handles available to a process?

Comment 13

[Jim Moe]

See also the comment about restarting vs. quit/start in #495911.

Comment 14

[Jim Moe]

Yes, this still happens. A day after having set all of the account the same, some of the accounts have reverted to basic default settings.

Comment 15

[Wayne Mery (:wsmwk)]

Jim, how many:
- imap accounts and total imap folders?
- pop accounts and total pop + local account folders?  any gmail?
- how many virtual folders aka saved searches? And do any one of these reference more than say 12 real folders?

Note, the default fd (open file) limit per process for Mac is 1,024. So probably we're hitting that.  Perhaps related to an addon (or perhaps not) that you mention in bug 495911 comment 40:
 - lightning
 - provider for google
 - test pilot for thunderbird
 - saved password editor
Please remove (not disable) them one by one, until all are gone, so we can determine whether any are involved.

(This bug is causing your address book opening problem, so we ignore bug 495911 until we settle this bug.)

Comment 16

[Wayne Mery (:wsmwk)]

Also, are you not seeing "Unable to open the summary file ..."?

(comment 8 is directly related to running out of fd.)

Comment 17

[Wayne Mery (:wsmwk)]

and p.s., are there any addons installed that you already have disabled?

Comment 18

[Jim Moe]

> - imap accounts
34
>  and total imap folders?
730

> - pop accounts and total pop + local account folders?
0

>   any gmail?
There are 3 Gmail IMAP accounts

> - how many virtual folders aka saved searches? 
>   And do any one of these reference more than say 12 real folders?
>
None.

There are no newsgroups.


> - lightning
> - provider for google
> - test pilot for thunderbird
> - saved password editor
  - also Folderpane Tools
> Please remove (not disable) them one by one, until all are gone, 
> so we can determine whether any are involved.
>
I have done this already. That is how I found a partial workaround. By using the "Restart" option of an en-/disabled addon, I can get the Address Books to load. No help for the settings, though.

It makes no difference for the address books or settings if any or none of the existing addons were en-/disabled. Or even removed.

Comment 19

[Jim Moe]

Does this mean this issue is being addressed?

Comment 20

[Jim Moe]

This bug has now infected the linux version, v24.2.0, linux v3.11.6-4-desktop x86_64. I have been cursing my way through restoring my settings for a random selection of folders that have had their settings reverted to program default.

If all it takes is to increase the file handle requirement, why hasn't it been done?

Comment 21

[Jim Moe]

Still an issue in v24.3.0 for linux and MacOS. This is a deeply annoying problem with a simple fix.

Comment 22

[Jim Moe]

[[e-mail]] wrote:
>> This is a deeply annoying problem with a simple fix.
> please offer your fix so that all will benefit.
>
I am presuming this issue and https://bugzilla.mozilla.org/show_bug.cgi?id=495911 are caused be the same problem: Insufficient file handles during startup. Once the number of accounts and folders exceed some number (somewhere around 500 - 700), this issue and the Address Book issue arise.

Comment 23

[Wayne Mery (:wsmwk)]

(In reply to Jim Moe from comment #22)
> [[e-mail]] wrote:
> >> This is a deeply annoying problem with a simple fix.
> > please offer your fix so that all will benefit.
> >
> I am presuming this issue and
> https://bugzilla.mozilla.org/show_bug.cgi?id=495911 are caused be the same
> problem: Insufficient file handles during startup. Once the number of
> accounts and folders exceed some number (somewhere around 500 - 700), this
> issue and the Address Book issue arise.

We can't assume the cause on a windows system is the same. open files limits are different on different OS.

However, iirc, Mac and linux have similar OS and per process max open fd limits.

In my experience the problem typically is not Thunderbird, but something in the user's environment. For example see bug 495911 comment 34 for a typically Mac problem.

On Mac you can use lsof to determine how many files are open system-wide, and also how many fd, and what files, are open for the Thunderbird process.  Compare the list and values you find when Thunderbird, versus when Thunderbird is failing, and we will likely narrow the problem for you.


>>  - also Folderpane Tools
>> Please remove (not disable) them one by one, until all are gone, 
>> so we can determine whether any are involved.
>>
>I have done this already. That is how I found a partial workaround. By using the 
>"Restart" option of an en-/disabled addon, I can get the Address Books to load. 

For example, I would test this scenarios with lsof

> No help for the settings, though.
what settings?

Comment 24

[Wayne Mery (:wsmwk)]

p.s. 

> If all it takes is to increase the file handle requirement, why hasn't it been
done?

because file handle limits are on user's systems for a reason.
However, iirc, this idea was floated in a mozilla core font bug report.
(I can't remember the bug # right now)

Comment 25

[Jim Moe]

>> No help for the settings, though.
> what settings?
The subject of this report: Randomly resetting views (sort order, threading) and display settings to default values.

> you can use lsof to determine how many files are open...
lsof results: open files
MacOS
quit   : 653     start with no Address Book error
restart: 826

quit   : 675     start with Address Book error
restart: 810

linux
quit:    477
restart: 425

That is a lot of files per process. The number varies a bit, +/-30, per invocation. Comparing file lists does not seem an effective method, especially since the quit/restart difference for MacOS is 150 files.

Comment 26

[Wayne Mery (:wsmwk)]

(In reply to Jim Moe from comment #25)
> > you can use lsof to determine how many files are open...
> lsof results: open files
> MacOS
> quit   : 653     start with no Address Book error
> restart: 826
> 
> quit   : 675     start with Address Book error
> restart: 810

that's rather interesting


> linux
> quit:    477
> restart: 425
> 
> That is a lot of files per process. The number varies a bit, +/-30, per
> invocation. Comparing file lists does not seem an effective method,
> especially since the quit/restart difference for MacOS is 150 files.

lsof lists file descriptors, which is way more than just Thunderbird files. It includes tcp connections for example. And, you may recall I mentioned fonts are also open.

please grep lsof for something like "Library/Thunderbird/Profiles" to get the "files" that thunderbird has open

Comment 27

[Jim Moe]

> please grep lsof for something like "Library/Thunderbird/Profiles"
Before filtering for "Library/Thunderbird/Profiles":
 wc -l lsof?.txt
   671 lsof2.txt    - Quit/strat
   835 lsof3.txt    - Restart

After filtering for "Library/Thunderbird/Profiles":
 wc -l lsof*-t.txt
   266 lsof2-t.txt    - Quit/start
   417 lsof3-t.txt    - Restart

I then removed some of the extraneous columns (FD, TYPE, DEVICE, SIZE/OFF), and sorted the results. The primary difference between the two data sets is that lsof3-t.txt has a lot more *.msf files open.

I would upload the files but there does not appear any way to do so.

Comment 28

[Jim Moe]

Hmm. Gone dormant again?

Comment 29

[Jim Moe]

Is there any hope of any action actually happening on this issue?

It is an PITA to have to randomly (and usually daily) re-set my preferred viewing settings for whichever folders lose the file handle battle. Other users here are tired of having to use the "restart" workaround to get access to their Address Books.

Comment 30

[WADA:World Anti-bad-Duping Agency]

Jim Moe(bug opener), can get Opensnoop log for Tb's file access during restart and check "actually consumed file handles by Tb"?
> https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/fs_usage.1.html
> https://developer.apple.com/library/mac/documentation/Darwin/Reference/Manpages/man1/opensnoop.1m.html
To know accurate timestamp when error message was shown, get NSPR log at same time, please.
> Win example : SET NSPR_LOG_MODULES=timestamp,DOMLeak:5,DocumentLeak:5,nsDocShellLeak:5
> window/panel open/, load document to window/panel is logged. So timestamp of dialog is known by log.

Comment 31

[Jim Moe]

> can get Opensnoop log for Tb's file access during restart 
> and check "actually consumed file handles by Tb"?

Check what where?

I ran opensnoop twice, once for TB starting, and once for TB restarting. The results are consistent with the data in comment 27.
$ ./opensnoop -n thunderbird > tb-file-list-xxx.txt

$ wc -l tb-file-list-*
  2191 tb-file-list-restart.txt
  1675 tb-file-list-start.txt

I would upload the data files except there does not appear to be any way to do so.

What is NSPR? Where is the log located?
Since this is a MacOS, I am not clear what good a Windows setting is.

Comment 32

[WADA:World Anti-bad-Duping Agency]

See bug 402793 comment #28 and read pointed documment. Thenere is no difference amnong OS in environment variable name and its data. Merely "used command to set environment variable" depends OS.

> The results are consistent with the data in comment 27.
I see.
Following is alredy found by you.
- While restarting Tb, Tb openes many *.msf files
- After it, 
  (a) when many folders are defined or files for folder exists,
      file open of xyz-N.mab fails, and error message is shown
  (b) when number of folders or files for folder is not so large,
      "number of file open log" is smaller than (a),
      and error message of "file open of xyz-N.mab failed" is not shown.

(Q1) In (a), is "address book which uses xyz-N.mab" accessble after end of restart?
(Q2) When error message is shown, is error message shown for other zzz-N.mab file?
(Q3) What kind error is logged for "open error of xyz-N.mab" in your monitor log

FYI. If abook-N.mab is opened using "Open Write" call by otherr software on Win-XP such as batch type script, error is "Sharing Violation", and once error occurs, access of Addressbook which uses abook-N.mab was imposibble until restart of Tb, because it's never shown address book list, even after other software closed the abook-N.mab file.

Phenomenon of "error message on address book file" is merely following, isn't it?

1. If many file for mail folder is used by Tb, many files for mail folder are opened by Tb.
   "number of opened file" is proprtional to "number of folders in Tb".
   So "required number of file hadle" is also "number of folders in Tb".
2. If the "required number of file hadle" exceeds, "file open by Tb" fails sooner or later.
3. If "file open failure during restart due to file handle shortage" occurs on xyz-N.mab file,
   error message is shown by Tb.
   If "file open failure during restart due to file handle shortage" occurs on x.msf file,
   error message is not shown by Tb.
4. "open x.msf during restart of Tb" is a result of folder re-discovery process.
   for example. 1. delete panacea,dat, 2. delete all *.msf files, 3. restart Tb
   => Tb scans all mail directorieas and subdirectories, and creates xxxx.msf if x.msf doesn't exist
   This is not process like next;
      for(each file for mail folder){open file/file.msf;access file/file.msf;close file/file.msf;}
   This is usually action like next.
     Kick a folder re-discover task for each account.
     A folder re-discovery task does folder re-discovery work recurively.
     function A_Task_Job(FileName){
       check FileName and FileName.msf => open Filename.msf occurs
       for(each FolderName+".sbd" directory){
          for(each file in each FolderName+".sbd" directory){ A_Task_Job(file); }
          close FolderName+".sbd" directory
       }
       close FileName;
     }
   If this kind of task, "number of required file handle for this job" > 1,
   so "number of required file handle for this kind job" is never 1.
5. This ".msf file open during restart of Tb" is never ordinal "MsgDatabase Open"folder open).
   If ordinal "MsgDatabase Open" for explit folder open(folder click at folder pabe),
   error message is shown for "*.msf file".
   1. when xxx.msf is not opened(folder xxx is not opened), "Open Write xxx.msf" by other software.
   2. Click folder xxx at folder xxx at folder pane.
   => pretty similar message to "abook.mab case" is shown for xxx.msf file.
6. Difference between Linux and Mac OS X.
   - Linux   : Doesn't keep font file open
   - Mac OS  : Does keep font file open
               i.e. your file handle limit setting in Mac OS X is too small for your usage
               See bug 495911 comment #108

Symptom is merely ;
- Your "file handle limit" setting is smaller than "actually required number of file handles during
  restart of Tb under an OS", so "file open failure due to file handle shortage" occurs
  during restart of your Tb.
- When error on *.mab file, error message is shown by Tb.
  When error on *.msf file, error message is not shown by Tb.
Wrong? If so, what part is wrong?
Excess file handle consumption by Tb during restart of Tb?
If "Excess", how excees? What is correct or valid or proper "number of file handles consumed by Tb during restart of Tb"?

Comment 33

[WADA:World Anti-bad-Duping Agency]

If simple symptom which I assumed in previus comment,problem should occur consistently, in any restart #2 to restart #N in next steps, on same or similar xyz-N.mab file, because relevant files are unchaged by shutdown/restart.
(1) Copy many files under Tb's mail directory, restart Tb #1
(2 to N) Terminate Tb normally Restart Tb #2 to #N.

Do you see your problem consistently in Restart ##2 to #N of above steps?

Comment 34

[WADA:World Anti-bad-Duping Agency]

(In reply to Jim Moe from comment #8)
> This issue appears to be related to this error message that shows in the
> Error Console:
>(snip)

SmartFolders==Unified Folder. All Unified Folder==A Virtual Folder==A Search Folder"\ is always opened at restart of Tb once, and is closed after while based on MsgDB idle time threshold, and is opned when HdrAdded/HdrDeleted event happened at search target folder, and both Virtual Folder itself and all search target folders are opened upon explicit Folder Open by click of the Virtual at folder pane.
Do you use View/Folders/Unified? If no, "explicit Folder Open by click of the Virtual at folder pane" part is irrelevant.

> Timestamp: 6/7/13 2:17:49 PM
> Error: [Exception... "Component returned failure code: 0x80004003
> (NS_ERROR_INVALID_POINTER) [nsIMsgFolder.getStringProperty]" nsresult:
> "0x80004003 (NS_ERROR_INVALID_POINTER)"  location: "JS frame ::
> chrome://messenger/content/folderPane.js :: getSmartFolderName :: line 2436" data: no]
> Source File: chrome://messenger/content/folderPane.js Line: 2438
> It shows dozens of times, grouped by regular intervals. It occurs at the
> same time that the folder appearance sometimes spontaneously reverts to
> default values.

Recent code is http://mxr.mozilla.org/comm-central/source/mail/base/content/folderPane.js#2389
> 2391     return aFolder.getStringProperty("smartFolderName");
aFolder corresponds to msgFolder=="a pluggable msgStore"==xxx file if xxx folder, and sFolder.msgDatabase corresponds to xxx.msf file. IIRC, although getStringProperty() is object method of msgFolder, StringProperty is saved in msgDatabase(=xxx.msf), because setStringProperty() is object method of msgDatabase.
If special folder such as Inbox, Trash, Drafts, ..., associated Unified Folder exists. And associated Unified Folder name is saved StringProperty. To check "whether assciated Unified Folder exists or not", to get "associated Unified Folder name", aFolder.msgDatabe is perhaps accessed by  aFolder.getStringProperty() call. If "file open of xxx.msf" fails due to file handle shortage, "file handle of xxx.msf" is perhaps set as null.
I think this is ".msf file version" of "The file XXXX could not be opened. ... xyz-N.mab" during restart.

Does this error stop after end of restart Tb?
Or repeatedly shown upon, for example, Drafts folder click, Sent Folder click, Junk folder click etc.?

Comment 35

[WADA:World Anti-bad-Duping Agency]

Because you define pop3 account imap account but you don't define nntp account, all file used for mail folder is held under .../<A_ProfileDirectory_Of_Tb>/Mail and .../<A_ProfileDirectory_Of_Tb>/ImapMail, if you doesn't alter Server Setting/Local Directory: setting after account creation in Tb.
How many files/directoriesfor mail folder exists under these two directories?
  Tb's file file directory structure for mail folde :
    FolderX     : msgStore file, corresponds to msgFolder
    FolderX.msf : summary file,  corresponds to msgDatabase
    Folder.sbd directory : contaiaer of Sub/Sub.msf for subfolder named Sub under FolderX
  => SubSub under Sub under FolderX : .../FolderX.sbd/Sub.sbd/SubSub.msf and SubSub is used
If all corresponding xxx.msf file to xxx file is normally created, "# of mail folder for Tb" is "# of file ends with .msf under .../Mail directory and .../ImapMail directory" in your case.

Do you have excess files/directories under the .../Mail or /ImapMail directory? (for example, temp file for batch, directory for temp file for batch, output data of batch, ...)
Do you see set of file directory like next?
   abc.def.ghi     : directory
   abc.def.ghi.msf : file

Comment 36

[WADA:World Anti-bad-Duping Agency]

(In reply to Jim Moe from comment #25)
> lsof results: open files
> MacOS (Case-1/Case-2 is by me)
> Case-1
> quit   : 653     start with no Address Book error
> restart: 826
> Case-2
> quit   : 675     start with Address Book error
> restart: 810

I guess result is following.
> Case-0
> quit   : less than 653     start with no Address Book error,
>    and Exception... " ...: 0x80004003 (NS_ERROR_INVALID_POINTER) [nsIMsgFolder.getStringProperty]"
>    is not reported on any folder
> Case-1
> quit   : 653     start with no Address Book error,
>    but Exception... " ...: 0x80004003 (NS_ERROR_INVALID_POINTER) [nsIMsgFolder.getStringProperty]"
>    is reported on many or some folders
Right?

Comment 37

[WADA:World Anti-bad-Duping Agency]

Additional question about mail folder files.
Do you see "xxx.msf file which doesn't have associated xxx file" under .../Mail directory?
This kind of file may cause "exess file open during restart".

About panacea.dat content.
Can you attach panacea.dat file under Tb's profile directory?
Before attach file to this bug, check file content by Text Editor, and if confidential data or private data is contained, remove or replace them, please.
(Folder Name is saved in panacea.dat, and it can be confidential data or private data. If such case, select a data block such as data for Inbox, Drafts, Sent folder etc. only. Purpose is to see persistentDescripter part in panacea.dat data on Mac OS X)

Comment 38

[WADA:World Anti-bad-Duping Agency]

Oh, you can test on Linux. Check panacea.dat file content of Tb on Linux, please.
What is difference between panacea.dat on Mac OS X?
(if Linux, IIUC, data is "full path of file for folder" + "folder name", as on Win)

Comment 39

[Jim Moe]

A clarification: It is when TB is restarted from an addon, "Restart Now" or a "warm start," that there seems to be enough system resources to open the number of files required. When starting after having closed/quit TB, a "cold start," there is NOT enough system resources to the open files.

<panacea,dat>:
- I cannot find "persistentDescripter" in either the linux or Mac version of the file. Otherwise, I have not a clue what you wish me to find in that file.
- The file sizes are quite different: linux, 717KB; Mac, 3.1MB. The size increases with session time.
- As previously noted, I cannot attach files.

File count in ImapMail and Mail folders:
Mail: 9
ImapMail: 87

Comment 40

[WADA:World Anti-bad-Duping Agency]

Followin is a part of panacea.dat of Tb on Win.
Full path of .msf file, and folder name(8D=ABC part), is held in a row of table.
>   (8B
>    =C:\\Documents and Settings\\wada\\Application Data\\Thunderbird\\Profiles\
\\wkeci8t7.ZZZ\\Mail\\Local Folders\\ABC.msf)(9613=40004604)(8D=ABC)
>   (682=-1)(11B19=1396306318
No such full path part in panacea.dat on Linux?
persistentDescripter I called is "something on Mac OS X which corresponds to full path part in panacea.dat on Win", which is perhaps called "Persistent Descripter" in Mac OS X, which is an identifire of file(or directory, or other resource) assigned by Mac OS X. 

> File count in ImapMail and Mail folders:
> Mail: 9
> ImapMail: 87
Following is your comment #18.  
> > - imap accounts
> 34
> >  and total imap folders?
> 730
Why 34 files even though 730 folders?
Please note;
  .../Inbox.sbd/ABC.sbd/DEF.sbd/GHI.sbd/ ... / XYZ and XYZ.msf is also folder file of Tb.
  in Mac OS X, there two kinds of file : File in Finder, File in OS file system.
  I don't know which of "File in Finder" and "File in OS file system" is "file in Tb on Mac oS X".

> A clarification: It is when TB is restarted from an addon, "Restart Now" or a "warm start,"
> that there seems to be enough system resources to open the number of files required.
> When starting after having closed/quit TB, a "cold start," there is NOT enough system resources to the open files.

I prefer; 
  Restart Now                      : Emergency Restart
  Normal shutdown restart          : Normal or Warm Restart
  Restart after delete panacea.dat, Restart after copy/move profile directory
                                   : Cold Start

If closed/quit Tb, "OS's resource expanding speed" < "Tb's file open request rate"?
When close/quit Tb, was Tb surely/completely terminated?
  - If Error Console is opened and all other windows are cloded,
    "Start Tb" after it adds messenger window to existent Tb process.
  - If Tb's process still running for temination process after Tb's window on desktop is closed,
    "Start Tb" after it adds messenger window to still running Tb process.
    This is similar to "Error Console only + Start Tb".

Comment 41

[Jim Moe]

Is this what you want?
<(80
    =/home/jmoe/.thunderbird/8zuqvpp0.default/ImapMail/imap.sma-2.com/INBOX.ms\
f)(81=83014)(1A427A=187)(12D=2)(84=0)(1A4671=82e239f)(86=ISO-8859-1)
  (1666E8=200042)(1BD=2f)(88=INBOX)(6B7=2ff)(1918E2=189)(1A4242=198)
  (35F70=Sun Jul 10 10:25:42 2011)(1A4243=31b2f)(1A464E=1396417604)
  (A7=1)(19282D=1390543154)(89

Is this better?
$  ll --recursive ImapMail/ | grep .msf | wc -l
1482

Comment 42

[Jim Moe]

And a similar wad of text from the Mac panacea.dat:
<(80
    =/home/sma-user3/.thunderbird/tmmwd5ff.default/ImapMail/imap.sma.com/INBOX\
.msf)(21A42=80083014)(3C26D=a9)(123=2)(81=0)(37014=2958272b)(3C28B=c0b239)
  (82=ISO-8859-1)(15D50=41)(A1=2f)(84=INBOX)(11C=2ff)(553F=bc)(3C182=90)
  (3C231=263b6)(218B4=4bbe5e2d)(3C2B1=1291307558)(3C294
    =Thu Dec 02 09:17:14 2010)(86

Also on the iMAC:
# ll -R ImapMail/ | grep .msf | wc -l
     710

Comment 43

[WADA:World Anti-bad-Duping Agency]

Thanks for panacea.dat data.
- "absolute path + folder name" is comon among Win/Linux/Mac OS X
- "absolure path in panacea.dat on Mac OS X" is also full path in file syste of OS.
ll result. How abut Mail directory?
  Because mail directory for pop3, Local Folders, smartmail boxes(Unified folder), feeds
  is held under Mail, "# of .msf file under Mail" + "# of .msf file under ImapMail" == # of folders

Comment 44

[WADA:World Anti-bad-Duping Agency]

If you have back up of opensnop log, check log data, please.
(1) Find an open error log of xxx.mab for which error message was shown
    (open error surely happenedd)
(2) Find all logs for same "open error reason code".
    I guess "reeason code" is same between .mab and .msf.
(3) Sort by timestamp. 
    When error started to occur on .msf? 
    When error started to occur on .mab? 
    When error stopped to occur?
    File is accessed in alphabetic order of full path? Or randomly accessed?

Comment 45

[Jim Moe]

Comment 43: How abut Mail directory?
Neither TB installation has a POP3 accounts.

Comment 44: There are no error messages in the opensnoop logs. Do you have an email address where I can send these files? It will go faster than this back and forth.

Comment 46

[Jim Moe]

Today many of the folders (about 2/3) had the Sort and Thread preferences reset to default.

Comment 47

[WADA:World Anti-bad-Duping Agency]

(In reply to Jim Moe from comment #45)
> Comment 44: There are no error messages in the opensnoop logs. 
> Do you have an email address where I can send these files? It will go faster than this back and forth.
Sorry bu I don't have Mac, and I'm not familiar with opensnoop and his log. Check log content by yourself, please.

Comment 48

[WADA:World Anti-bad-Duping Agency]

(In reply to Jim Moe from comment #46)
> Today many of the folders (about 2/3) had the Sort and Thread preferences reset to default.
What do you call "Thread preferences"? Column choice at Thread Pane?
If yes, it's pretty natueal phenomenon when "write open of xxx.msf for IMAP folder" failed.
  "Column choice at Thread Pane" is saved in xxx.msf file.
  If "write open of xxx.msf" occurs, it's same as "delete xxx.msf" or "Repair Folder"
  from perspective of "xxx.msf file content".
  If IMAP folder, "Column choice at Thread Pane" is not recovered from xxx.msf content when "Repair
  Folder" is executed. This is already known bug.
  This is applicable to "sort order of a thread column" too.

Comment 49

[Jim Moe]

WADA: Why ask for data that you cannot process? I do not know what is relevant in the logs.

Has any of this helped to move toward a solution?

From the Menu::View::Sort By, my preferred settings are:
- sort by Date
- sort Descending order
- sort Threaded

"Sort by Date" appears to be the default so that never changes. The 2nd and 3rd preference are lost at random intervals.

On linux the column preferences are rarely lost. On MacOS (OS X 10.9 and earlier) both the sort and column preferences are lost fairly quickly, usually in less than a week.

Comment 50

[Jim Moe]

I forget to mention: I am now using v24.4.0. There is no change in the bad behavior.

Comment 51

[WADA:World Anti-bad-Duping Agency]

(In reply to Jim Moe from comment #49)
> WADA: Why ask for data that you cannot process? I do not know what is relevant in the logs.

Your report is currently;
 (i) With max file handle = FFF,
    (ii) When # of folder = N,
         (iii) it's unclear whether "exception PERHAPS on zzz.msf" occurred or not,
         (iv)  "file open error on xxx.mab" was not reported.
    (v)  When # of folder = (N+a),
         (vi)  "exception PERHAPS on zzz.msf" was reported in Error Console.
         (vii) "file open error on xxx.mab" was reported.
This should be changed to;
  Tb requested file open in this order, ...,  during restart of Tb.
  "File open error" occured on file named f1, f2, ... at t1, t2, ...
  "Exception" reported at tx1, tx2, ... by Tb
  "File open error" occured on xxx.mab at ty1, and "file open error on xxx.mab" was reported by Tb

"File access monitor log" usually has at least following data on any OS.
  Timestamp, Process name of requester, File path,
  Request(open, close, read, write, ...), Result(success, failure, Reason code if failure)
You can't see such data in your opensnoop log? Opensnoop won't report such information?

Comment 52

[WADA:World Anti-bad-Duping Agency]

(In reply to Jim Moe from comment #49)
> On linux the column preferences are rarely lost. On MacOS (OS X 10.9 and
> earlier) both the sort and column preferences are lost fairly quickly,　usually in less than a week.
I guess "xxx.msf file open error due to file hangle shortage during restart" occurs even when "yyy.mab file open error" doesn't occur and Tb doesn't show error message.
For Tb, "xxx.msf file open error durin folder re-discover" is similar to "deleted xxx.msf", rom perspetive of "reading xxx.msf content for MsgDabase open". So, same phenomenon as "Repair Folder" occurs on Thread Pane Column Header if IMAP folder. (see bug 550286)

Do you see exception which looks relevant to Message Folder or MsgDatabase in Error Console?

Comment 53

[WADA:World Anti-bad-Duping Agency]

FYI.
Bug 717574 is for similar "xxx.msf file open error" message upon MsgDatabase open for ordinal folder open by Virtual Folder, when vary many search target folders are defined for a Search Folder(Virtual Folder).
In this case, folders are opened by code like "for(Folder in search target){open Folder;)". So, for file open, "for(each xxx.msf file for Folder){open write xxx.msf;read xxx.msf;}" like code is executed. And, "close xxx.msf" is executed later based on MsgDB idle time threshold and limit of # of opened MsgDBs. So, many xxx.msf files are kept open for relatively long time.
In this case, "sufficient number of available file handles" is mandatory.
In your case, I believe following is a workaround.
  Define Virtual Folder, with setting all your mail folders as "search target folder".
  Increase "max file handle" until the Virtual Folder is opened without "open error of search target".

Comment 54

[Jim Moe]

WADA: Are you a developer or a maintainer?

Virtual Folder: Useless for me since the two OSes of interest here are OSX and linux. VF is for Windows only.

Comment 55

[WADA:World Anti-bad-Duping Agency]

(In reply to Jim Moe from comment #54)
> WADA: Are you a developer or a maintainer?
No. I merely a helper of developers. QA work, code testing for developers, is my interests in B.M.O.
I'm never trying to help general Tb users at B.M.O.

> Virtual Folder: Useless for me since the two OSes of interest here are OSX and linux. VF is for Windows only.
VF is OS independent feature, and is merely a "Saved Search" currently. Biggest problem in VF is: Over expectation on VF by users. VF is very convenient, because it's usable as excellent　"outdated msf condition" detector and "Rebuild-Index starter" for many folders which have "outdated msf condition". See bug 905576. VF can be used also as tool to find "required number of file handles".

Comment 56

[Jim Moe]

Another much more serious consequence of this issue is the permanent loss of messages.

I recently discovered that one of my primary accounts had a default retention policy of 30 days (I have no idea why). Guess what? The preferences were reset to default on that account and I lost a bunch of messages.

Comment 57

[Jim Moe]

v24.5.0.

Please, please, PLEASE fix this! (On knees, folded hands, tears in my eyes.) 

Yet again the message display preferences were reset to default in an account. Which means having set the preferences in dozens of folders.

Comment 58

[Wayne Mery (:wsmwk)]

Hi Joe. Thank you for continuing to supply informatin.  Let's attack this again.

Is version 17 the first version where you encountered this issue? 
And if you install version 16 from http://download.cdn.mozilla.net/pub/mozilla.org/thunderbird/releases/16.0.2/mac/en-US/ the problem does not occur?

Can you please do an NSPR log previously requested, but as follows
  SET NSPR_LOG_MODULES=timestamp,DOMLeak:5,DocumentLeak:5,nsDocShellLeak:5,msgdb:5
  see https://wiki.mozilla.org/MailNews:Logging


(In reply to Jim Moe from comment #56)
> Another much more serious consequence of this issue is the permanent loss of
> messages.
> 
> I recently discovered that one of my primary accounts had a default
> retention policy of 30 days (I have no idea why). Guess what? The
> preferences were reset to default on that account and I lost a bunch of
> messages.

Yes, this could be very bad. a consequence of retention settings being saved in .msf files that go corrupt, lost or otherwise.

Comment 59

[Jim Moe]

I have an NSPR log. Where do I send it? I do not see any way to attach or otherwise upload it.

Comment 60

[WADA:World Anti-bad-Duping Agency]

(In reply to Jim Moe from comment #59)
> I do not see any way to attach or otherwise upload it.

Link of following at top part of this bug is way to attach file to this bug. 
  Attachments
  <Add an attachment> (proposed patch, testcase, etc.)

Comment 61

[Jim Moe]

Attachment: imap.log

I have attached the log file using the following script:

export NSPR_LOG_MODULES=timestamp,DOMLeak:5,DocumentLeak:5,nsDocShellLeak:5,msgdb:5
export NSPR_LOG_FILE=$HOME/imap.log
/Applications/Thunderbird.app/Contents/MacOS/thunderbird-bin &

Comment 62

[Jim Moe]

Further on the attachment:

I started TB, checked that all of the address books loaded (they did), quit TB.

Comment 63

[Jim Moe]

TB 31.4.0
linux 3.16.7-7-desktop x86_64

Another clue. Or workaround, possibly.

Today, when I started TB, I opened an account long before it had been read from the server. It showed the process of being populated with all of its folders.

Along with the visual progress display, ALL of the folders were reset to default settings. This has not happened in a long time.

Apparently the process of loading lots and lots of folders with the display visually showing the population is part of the problem.

Comment 64

[WADA:World Anti-bad-Duping Agency]

(In reply to Jim Moe from comment #61)
> imap.log

Max "NN open DB's" looks "87 open DB's". As 12 x ".../<servername>.msf" is seen, perhaps 12 accouts8or more) are defined.

I can't imagine resone why file handle shortage can occur  by Tb's "87 open DB's" in your environment, if appropriaate file handle number limit is set in OS by you.
What will occur when small mail.db.max_open such as mail.db.max_open=5 is set? (default=30, so 360 DB can be opened at same time if 12 accounts)

Comment 65

[Wayne Mery (:wsmwk)]

This and bug 893786 should be tested with version 38

Comment 66

[Jim Moe]

I have tested TB v38.1.0 on OS/X 10.10.x (Darwin Kernel Version 14.4.0).

If anything, the problem is worse. I (yet again) set the preferences (accounts and message display) for each of the accounts and folders. After restarting TB, half of the accounts/folders' settings had reverted to default. Previously such a decay to default would take 1 - 3 weeks.

I would still like to have some to way to alter the default settings for the accounts and message display window. This would be useful as a workaround for this long-neglected issue, and would set new accounts and folders to my preferred settings without further action on my part.

Comment 67

[Jim Moe]

(In reply to WADA from comment #64)
> (In reply to Jim Moe from comment #61)
> > imap.log
> 
> I can't imagine reason why file handle shortage can occur
>
Nor can I. Nor am I at all certain file handle exhaustion is the cause of the issue. I only know the issue is real, that it is a giant PITA, that it is worse on OS/X than on linux, that there has been no remedial action since 2009 when it was reported in bug #495911.

Comment 68

[Jim Moe]

A status update:

The Address Book aspect has been resolved. It appears that is only loaded on demand now, rather than at start-up. Because it is loaded later, all address books are actually available.

The display preferences aspect, however, is noticeably worse; loading the ABs later has not made any difference. Now, even Tagging data can be lost along with all of the folder and message display preferences. It takes less than a week for TB 38.1 to completely revert to its default settings.

Comment 70

[Jim Moe]

This continues to be an issue on linux and OS/X.