Closed Bug 1046392 Opened 10 years ago Closed 10 years ago

Thunderbird 31 slow. Set to show only subscribed folders, but TB is using LIST to query folders

Categories

(MailNews Core :: Networking: IMAP, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 859269

People

(Reporter: dickie, Unassigned)

References

()

Details

(Keywords: perf, regression, Whiteboard: [regression:tb2?])

User Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24) Gecko/20111108 Fedora/3.6.24-1.fc14 Firefox/3.6.24 (Nightly/Aurora)
Build ID: 20111108102749

Steps to reproduce:

Accessed an IMAP account using Thunderbird 31.


Actual results:

Thunderbird displayed this message at the bottom of the main mail window "Looking for folders..." and there is a blue spinning wheel. I cannot access my IMAP Inbox nor any of my folders.


Expected results:

Thunderbird should display the current IMAP account Inbox and folders and
allow access to them.
I have seen this hang on my Mac OS X 10.8.5 system.

My co-workers are seeing the same hang on their Linux and Windows laptops running Thunderbird 31 when accessing their IMAP accounts. One co-worker found that if he changes Thunderbird to not subscribe to folders the hang is resolved. This is only a work-around and not a suitable solution in general because now every item in their folder, mail or not, is displayed by Thunderbird.
Note that reverting back to Thunderbird 24.5 fixes this problem.

I talked with yet another co-worker who is also having problems updating to Thunderbird 31 with his IMAP mail account. He turned on debugging and found that Thunderbird is now accessing the IMAP server using a different command to access the user's root mail account. This IMAP command causes the IMAP server to chase symbolic and hard links in our NFS mounted home directories that leads to potentially hours of elapsed time to complete.

Apparently there are serious performance implications using this IMAP command that causes these kinds of performance problems that appear as hangs. He is trying to find his debug logs from Thunderbird to find the exact IMAP command causing this problem in Thunderbird 31. Previous releases of Thunderbird use a different command to access the IMAP account.
I'm the original reporter's coworker, and I can provide some information regarding our configuration and what I've seen going wrong via Thunderbird's IMAP debug logs.

Setup background: Our IMAP server is running Cyrus IMAP. This server is also our NFS home directory server, and as such has full access to not only email folders, but all our home directories and files as well.

The key problem we're observing is how Thunderbird is attempting to enumerate user folders. This behavior is different between whether Thunderbird's "Show only subscribed folders" option is selected.

When this option is selected, it appears that Thunderbird uses an IMAP "LIST *" command to recursively list all folders. On our IMAP server this list is rooted at the user's home directory, and as such begins recursing the subdirectories looking for files, which it does find in great abundance. This ends up being particularly bad for us as we extensively use NFS in our environment, and (at least in my case) my home directly structure contains symlinks to large directory hierarchies (tens of thousands of files) contained on other NFS servers, so the IMAP server ends up recursing every single one of those files. To make matters even worse, some of those directories themselves contain symlinks higher up in the NFS filesystems (e.g. to the root of the share), so the IMAP server ends up also recursing all the directories of all the other users on the NFS share -- there are literally tens of millions of files reachable by this recursion.

When this option is not selected, Thunderbird instead uses a "LIST INBOX.%" command to discover folders. This command ends up listing only the INBOX folders, and does so non-recursively. Thus we don't see the problem presented above.

Note that I have run across descriptions of the IMAP protocol where they advise against using "LIST *" commands due to performance. A good example of such advice can be found at: http://www.skytale.net/blog/archives/23 ... -IMAP.html

This behavior was determined by obtaining logs by setting the following environment variables:

NSPR_LOG_MODULES=IMAP:5
NSPR_LOG_FILE=/tmp/thunderbird.log

The resulting log files have the following key lines in them (server details obscured):

Failing case (allow view of all folders):
963639040[7f413c14e1e0]: 39f9a800:[server name]:A:SendData: 6 list "" "*"

Working case (only view subscribed folders):
-1411385600[7f6fae2c2080]: ac0c2000:[server name]:A:SendData: 7 list "" "INBOX.%"

To our knowledge this behavior is new in Thundebird 31. I was previously not a Thunderbird user, and was recently trying to make the switch from alpine/pine/re-alpine to Thunderbird. My Thunderbird configuration worked for a time (approximately 2 days), and then (presumably when the system auto-updated to Thunderbird 31) it stopped working and I dived into the debugging, figuring that I must have messed something up as a new Thunderbird user. I "started fresh" by removing all my Thunderbird configuration files many times during these debugging sessions, so I know that it's not some stale setting causing this problem. I can confirm that pine/alpine/re-alpine do not demonstrate this problem.

I also asked our IMAP/NFS server admin whether there had been any recent changes to the IMAP configuration, to which she replied that there had been no recent changes (neither configuration nor software update), and that the configuration had been stable for years. So this does not appear to be a server configuration or IMAP server software problem. The problem really does appear unique to Thunderbird, and as far as we can tell Thunderbird 31 in particular.
Dick, I've clarified the bug summary to hopefully more closely match what you are seeing. Please edit further if needed.

If we are lucky irving may have an idea of where to go hunting. If not then  it would be a great help for you to narrow the regresion range to find which beta stops working. We know 24 works and 31 fails. You'd want to do a binary search of the beta builds 25-30 at http://download.cdn.mozilla.net/pub/mozilla.org/thunderbird/releases/ 

BTW, http://www.skytale.net/blog/archives/23 isn't a helpful link

In case any of these might be related...
https://bugzilla.mozilla.org/buglist.cgi?v4=imap&f1=short_desc&o3=anywords&list_id=10880051&v3=imap&o1=nowordssubstr&resolution=---&resolution=FIXED&classification=Client%20Software&classification=Components&f4=short_desc&query_format=advanced&f3=component&f2=OP&o4=substring&longdesc=LIST%20%2A&longdesc_type=substring&product=MailNews%20Core&product=Thunderbird
Bug List
https://bugzilla.mozilla.org/show_bug.cgi?id=635216
635216 – crash [@ nsImapProtocol::List(char const*, int, int)]
https://bugzilla.mozilla.org/show_bug.cgi?id=495318
495318 – add support for LIST (SUBSCRIBED) part of IMAP LIST-EXTENDED command (RFC 5258)
https://bugzilla.mozilla.org/show_bug.cgi?id=885162
885162 – Nonexistent IMAP folders listed not just in the folder pane but also in a refreshed subscribe window without check box for (un)subscribe (After "LIST (subscribed)" support by Bug 495318, there is no way in Tb to unsubscribe \NonExistent \Subscribed Mbox)
https://bugzilla.mozilla.org/show_bug.cgi?id=859269
859269 – upgrading to 20.0b1 hangs on "looking for folders" on uw-imap server (Bad configuration of an UW-IMAP server was unfortunately exposed by unplesant change to 'LIST "" "*" use again' by bug 799821)
https://bugzilla.mozilla.org/show_bug.cgi?id=799821
799821 – Folders misbehave when LSUB does not return mailbox flags
https://bugzilla.mozilla.org/show_bug.cgi?id=918968
918968 – shared imaps folders only show synchronized contents but are readonly and not updated
Flags: needinfo?(dickie)
Keywords: perf, regression
Summary: Thunderbird 31 Hangs Accessing IMAP Account → Thunderbird 31 slow. Set to show only subscribed folders, but TB is using LIST to query folders
Whiteboard: [regression:tb2?]
Sorry, I have neglected fixing this. No one ever complained about it in any of the beta versions (25-31), so I never made it a priority. 

This is indeed bug 799821. UW-IMAP has the same sort of configuration that exposes the users entire home directory to LIST "" "*" and this causes the hangup.
Status: UNCONFIRMED → RESOLVED
Closed: 10 years ago
Resolution: --- → DUPLICATE
Please read over bug 799821 and we can pick up the discussion there on what is the best to resolve this.
Wayne and David,

Thanks for your help.

I have involved our company IT folks in this problem and they are available to help.

Let me know what I can do to further help;  and my coworkers are available to help too.

Dick
Flags: needinfo?(dickie)
I conform the same problem.

Thunderbird 31 hangs both on various Macs (under Maverick) and Windows 2003 server. It hangs in the 'Looking for folders' after startup, showing a round blue ball.

I have had this running during several hours, without help.

On the IMAP server side we have uw-imap-2007 on Redhat RHEL 6.

Returning to Thunderbird 24.6 or 24.7 solves the problem.

My mail folders on the server have no NFS involved.
The imapd process on the server seems to sleep, after an initial consumption of about 3 CPU minutes. On Windows, the Thunderbird client seems also to sleep.
I am surprised that status was changed to RESOLVED. 

It may be a duplicate of an older bug, but nothing happens there, at bug 799821.

The bug has become serious, because it affects now very many users after the public release of Thunderbird 31.
I think this bug should be kept separate, as the bug 799821 has a different subject.

Also here, the subject is wrong.

Bug 1046392 makes Thunderbird 31 completely unusable, not only slow. When it is opened, it never hangs in the 'looking for folders' stage, with several different and common IMAP servers.
I can confirm that with Thunderbird 32.01b (on Mac) the problem I had with version 31 has gone away.
I confirmed that Thunderbird 31.1 fixes the hang at start-up when accessing an IMAP mail account.
You need to log in before you can comment on or make changes to this bug.