Closed Bug 273742 Opened 20 years ago Closed 19 years ago

Thunderbird performance issues when fetching the headers for a very large IMAP mailbox

Categories

(Thunderbird :: General, defect)

Product:
Thunderbird
Component:
General
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED EXPIRED

People

(Reporter: thom, Assigned: mscott)

Details

(Keywords: perf) 

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0

The following test was performed at Sendmail, Inc. in response to a customer
problem relating to mailbox access and viewing. I would be glad to help further
test the issue, if you'd like. It's also possible that some option may be used
in Thunderbird to better handle a large INBOX - any thoughts are welcome.
Thanks. Short report below. -thom

IMAP retrieval testing with SAMS 2.1.5 on RHAS 3.0

Test:
The following test discusses the IMAP4 retrieval of 174,844 messages
in a single INBOX.

Background:
This test was required to determine why messages retrieved from a similar
mailbox by Mozilla Thunderbird or Microsoft Outlook took so long that
the INBOX is essentially unavailable.

Test Parameters:
- Sendmail Advanced Message Server 2.1.5 software serving IMAP4.
- RedHat AS 3.0 server, kernel 2.4.21-15.0.4.EL, 1GB RAM, two (2) 1266MHz CPUs
- mailstore partition is ~25GB on ext3, using a default filesystem create
  command:
     /dev/cciss/c0d1p1 on /data1 type ext3 (rw,noexec,nosuid,nodev)
- messages are of average size 32kB, ranging in size from 1486 bytes (1kB +
  headers) to 5301371 bytes (5MB + headers)
- Mozilla Thunderbird versions used to test:
    0.5.0 on Windows
    1.0 on Linux

Test Method:
1. Create INBOX containing 174,844 messages, using MailStone SMTP testtool
   (http://lxr.mozilla.org/mozilla/source/mstone/doc/MailStone.html). The
   total mailbox size tested is 6183296 bytes (~6GB)

2. Make a tar backup of inbox in order to "restore" untouched mailbox
   with each test.

3. Capture all IMAP data using the following command (IMAPD is running on
   port 2143 on the server):
   # tcpdump -i eth0 -s 0 -n -w /var/tmp/imap.dump-TESTNAME port 2143

4. Do a test connect with Thunderbird, and use tcpdump command to capture
   exact list of IMAP protocol commands. This is the list of commands
   used by the Thunderbird client:

1 capability
2 login "site-errors" "PASSWORD"
3 namespace
4 lsub "" "*"
5 lsub "" "Other Users/*"
6 lsub "" "Shared Folders/*"
7 list "" "INBOX"
8 list "" "Trash"
9 create "Trash"
10 subscribe "Trash"
11 list "" "Trash"
12 select "INBOX"
13 getacl "INBOX"
14 myrights "INBOX"
15 getquotaroot "INBOX"
16 UID fetch 1:* (FLAGS)
17 UID fetch 1:174844 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To
Cc Subject Date Message-ID Priority X-Priority References Newsgroups
In-Reply-To)])

after download of headers is complete:
14 check
15 noop
16 getquotaroot "INBOX"
17 UID fetch 174845:* (FLAGS)

5. Refresh mailbox (between each test), and run these tests:

    a. Test with Thunderbird on Windows
    b. Test with Thunderbird on Linux
    c. Test with netcat, piping output to file

6. The netcat command used to capture this data was:

    - in one shell window, do:
       # netcat fennel.ps-lab.sendmail.com 2143 > /var/tmp/imap-netcat.out
    - in other window, do:
       # tail -f /var/tmp/imap-netcat.out

   Then issue each of the above IMAP commands in succession.
   The netcat command is run on the exact same laptop as the Thunderbird
   1.0 for Linux was run, and on the exact same network.


Test Results:
=============================================================================
-----------------------------------------------------------------------------
                                         Command:
                   16 UID fetch 1:* (FLAGS)  |  17 UID fetch 1:174844 etc.
-----------------------------------------------------------------------------
                   Start      End     Total  |  Start       End      Total
 Test              Time      Time     Secs   |  Time       Time      Secs
-----------------------------------------------------------------------------
Thunderbird 0.5   09:13:40  09:13:44    4      09:13:44  14:29:47    18963
  on Windows

Thunderbird 1.0   09:14:54  09:14:57    3      09:14:58  11:13:16     7158
  on Linux

netcat on         12:21:46  12:21:47    1      12:22:02  12:25:38      216
  Linux
=============================================================================
tcpdump binary files can be provided for review. Uncompressed, each
file is over 70MB.

Conclusion:
It is apparent that Thunderbird is doing something inefficiently,
perhaps sorting the messages in the mailbox upon retrieval, or perhaps it
is not efficiently buffering the incoming data. This report will
be sent off to Mozilla team for further analysis (whether they respond
to it is out of our control, but we obviously hope they do). FWIW, Outlook
demonstrated similar (though worse) behaviour in production. The only client
test which could functionally work with this type of IMAP mailbox was pine or PC
pine.



Reproducible: Always
Steps to Reproduce:
Please see above.

Actual Results:  
Clock spins in Thunderbird for multiple hours.

Expected Results:  
Quickly displayed the list of message header data.
Thom, what does that test mailbox look like? For example, does every message
have the same subject? If that's the case, then you run into a pathalogical case
in our message threading code, which I need to fix. But if that's the case, I
can have you set a pref that avoids that problem on the client.
In my test example, the subjects look like this (as displayed in pine):

  N 174826 Sep 28 sender0@mycompany   (41K) 38K-mime benchmark message
  N 174827 Sep 28 sender0@mycompany   (11K) 10K-mime benchmark message
  N 174828 Sep 28 sender0@mycompany  (4612) 4K benchmark message
  N 174829 Sep 28 sender0@mycompany   (41K) 38K-mime benchmark message
  N 174830 Sep 28 sender0@mycompany   (11K) 10K benchmark message
  N 174831 Sep 28 sender0@mycompany   (11K) 10K-mime benchmark message
  N 174832 Sep 28 sender0@mycompany  (4629) 4K-mime benchmark message
  N 174833 Sep 28 sender0@mycompany  (8364) 7.6K-mime benchmark message
  N 174834 Sep 28 sender0@mycompany   (11K) 10K-mime benchmark message
  N 174835 Sep 28 sender0@mycompany  (4612) 4K benchmark message
  N 174836 Sep 28 sender0@mycompany   (41K) 38K-mime benchmark message
  N 174837 Sep 28 sender0@mycompany   (11K) 10K benchmark message
  N 174838 Sep 28 sender0@mycompany  (1512) 1K benchmark message
  N 174839 Sep 28 sender0@mycompany   (11K) 10K-mime benchmark message
  N 174840 Sep 28 sender0@mycompany   (41K) 38K-mime benchmark message
  N 174841 Sep 28 sender0@mycompany  (4629) 4K-mime benchmark message
  N 174842 Sep 28 sender0@mycompany   (11K) 10K-mime benchmark message
  N 174843 Sep 28 sender0@mycompany  (8364) 7.6K-mime benchmark message

In our customer's real INBOX, the message subjects would also have a lot
of similarity, as they are using the mailstore for the error messages
generated from all of their many webservers. Most of these error
messages are java exceptions, so there would be a lot of very similar
subjects.

so you end up with a lot of messages with the same subject? Can you try adding
the following to your prefs.js?

use_pref("mail.thread_without_re", false);

and re-run the tests? I am going to try to improve the performance in the case
of giant threads with the same subject, but it would be good to know if there
are other problems...
Keywords: perf
Testing with the following option in prefs.js:
   user_pref("mail.thread_without_re", false);

Test Results:
=============================================================================
-----------------------------------------------------------------------------
                                         Command:
                   16 UID fetch 1:* (FLAGS)  |  17 UID fetch 1:174844 etc.
-----------------------------------------------------------------------------
                   Start      End     Total  |  Start       End      Total
 Test              Time      Time     Secs   |  Time       Time      Secs
-----------------------------------------------------------------------------
Thunderbird 1.0   10:42:13  10:42:17    ~4     10:42:17   10:45:56    219
 on Linux with
 mail.thread_without_re=false
=============================================================================

So, it would appear that this is the problem. Very good, for everyone involved.

Perhaps you might want to consider allowing the default usage of Thunderbird to
be to just sort the INBOX in order of arrival (message UID). Then, only
thread/sort on other aspects when and if the user requests it.

Thanks very much for your assistance. Good luck,

-thom
This is an automated message, with ID "auto-resolve01".

This bug has had no comments for a long time. Statistically, we have found that
bug reports that have not been confirmed by a second user after three months are
highly unlikely to be the source of a fix to the code.

While your input is very important to us, our resources are limited and so we
are asking for your help in focussing our efforts. If you can still reproduce
this problem in the latest version of the product (see below for how to obtain a
copy) or, for feature requests, if it's not present in the latest version and
you still believe we should implement it, please visit the URL of this bug
(given at the top of this mail) and add a comment to that effect, giving more
reproduction information if you have it.

If it is not a problem any longer, you need take no action. If this bug is not
changed in any way in the next two weeks, it will be automatically resolved.
Thank you for your help in this matter.

The latest beta releases can be obtained from:
Firefox:     http://www.mozilla.org/projects/firefox/
Thunderbird: http://www.mozilla.org/products/thunderbird/releases/1.5beta1.html
Seamonkey:   http://www.mozilla.org/projects/seamonkey/

This bug has been automatically resolved after a period of inactivity (see above
comment). If anyone thinks this is incorrect, they should feel free to reopen it.
Status: UNCONFIRMED → RESOLVED
Closed: 19 years ago
Resolution: --- → EXPIRED
You need to log in before you can comment on or make changes to this bug.