420300 - Compact or Search folders hangs when there are more folders than the OS's MAXFILES

Status: RESOLVED DUPLICATE of bug 249754

(Thunderbird :: General, defect)

Description

[Jay Schuster]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12
Build Identifier: version 2.0.0.12 (20080213)

When I compact folders, Thunderbird exceeds my OS's open file limit, and then hangs.  I have over 600 IMAP mail folders (I'm sorry, but I've been using and categorizing my email since the 80s), and compact folders appears to leave a file descriptor open for each folder it compacts.

lsof shows each file being opened in turn, and never closed.  Here's a tail of it right now:

COMMAND   PID  USER   FD   TYPE    DEVICE SIZE/OFF     NODE NAME
...
thunderbi 928  jay  695u   REG      14,2     28237 13638183 /Users/jay/Library/Thunderbird/Profiles/0kxk8oq6.default/ImapMail/mail2.pcc.com/INBOX.sbd/vendors.sbd/ibm.msf
thunderbi 928  jay  696u   REG      14,2      9167 13638184 /Users/jay/Library/Thunderbird/Profiles/0kxk8oq6.default/ImapMail/mail2.pcc.com/INBOX.sbd/webmaster.msf
thunderbi 928  jay  697u   REG      14,2    108796 13638185 /Users/jay/Library/Thunderbird/Profiles/0kxk8oq6.default/ImapMail/mail2.pcc.com/INBOX.sbd/webmember.msf
thunderbi 928  jay  698u   REG      14,2      7839 13638186 /Users/jay/Library/Thunderbird/Profiles/0kxk8oq6.default/ImapMail/mail2.pcc.com/INBOX.sbd/www.msf
thunderbi 928  jay  703u   REG      14,2      1434 10899643 /Users/jay/Library/Thunderbird/Profiles/0kxk8oq6.default/Mail/pop.gmavt.net/Inbox.msf
thunderbi 928  jay  704u   REG      14,2      1941 10899646 /Users/jay/Library/Thunderbird/Profiles/0kxk8oq6.default/Mail/pop.gmavt.net/Trash.msf

Yes, that last one is file descriptor #704!

I've solved the problem by creating an /etc/launchd.conf that increases the number of open files from 256 to unlimited.  Not the smartest solution.

This bug also affects Search Folders.  This bug has been around since the early days of Thunderbird -- I could never get Search Folders or Compact Folders to work until I figured this out.

Reproducible: Always

Steps to Reproduce:
1. Create an account (or accounts) with more mail folders than your OS's limit on number of open files.
2. File -> Compact Folders  (or Search Folders)
3. Monitor it with lsof -- watch it get stuck when it hits the 255th (or whatever your OS limit is) folder.
Actual Results:  
Although Thunderbird doesn't become unresponsive when it hits MAXFILES, it seems to leave some of its internal data in an inconsistent state, and when I quit and restart it, it loses some of the properties on my IMAP folders (like check this folder for new messages).

When done by using `Search Folders', the search never completes.

Expected Results:  
I would expect the earch to complete, or compact folders to not corrupt its internal data structures and preserve the properties I've set on the folders.

Unless I bump MAXFILES on my computer (on MacOSX that means editing /etc/rc for Tiger and earlier; creating /etc/launchd.conf (or ~/.launchd.conf) for Leopard), Searches never complete.

Comment 1

[Melchert Fruitema]

Unless Thunderbird can check for this imposed limit on the number of open files, I would suggest a document change to address this limitation in the release notes for MacOS users.

Comment 2

[Jay Schuster]

The system functions getdtablesize() or getrlimit(RLIMIT_NOFILE, ...) will return the imposed limit.

Comment 3

[Wayne Mery (:wsmwk)]

bug 341647 describes this.
no dupes (that's surprising).
solutions?

Comment 4

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

I don't know much about this -- maybe bienvenu can help.

Comment 5

[David :Bienvenu]

yes, this is a dup of bug 249754, among others, probably. There's a patch in that bug, but I'm still trying to figure out if it's the right thing to do.

Comment 6

[Wayne Mery (:wsmwk)]

I only found a couple. But perhaps there is a better bug query than http://tinyurl.com/5hef3l

duping