214637 - IMAP access is slow in general [@ nsImapMailFolder::CopyMessages]

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Hans Wilmer]

User-Agent:       
Build Identifier: Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.5b) Gecko/20030723

The IMAP access is slow in general. Several reports concerning slow IMAP access
have already been added, but IMAP being very slow seems to be a general problem.

When starting the mail client in the morning, when a lot of messages have
arrived during the night, the client often crashes after accessing the INBOX and
has something like timeouts. During these timeouts, reading, deleting or
composing mail is difficult and can lead to crashes of the mail client. It seems
as if the client is somewhat blocked, busy to acquire the information about the
new mails. Closing the client and restarting it mott improves the performance.

But whenever you get to read a mail that is somewhat larger, it takes very long
to download and display the body. Especially when downloading large attachments,
it can take several minutes until the attachment is fully downloaded and can be
displayed.

My first assumption concerning these speed problems was that there´s maybe a
problem with server or network performance. But using other clients --- mutt (on
Linux) and becky on Windoze, I could verify that it takes only a few seconds to
download a large attachment, even while the mozilla client was trying to
download the same attachment from the same mail since some minutes.

The slowlyness is very annoying and appears on all versions of Windoze, i. e. 95
to 2000. It seems that the Linux version of Mozilla is not affected, but I can´t
tell that for sure because network environment I´m running the Linux version is
totally different.

Mail is accessed via LAN, the IMAP server in use is Cyrus, as it comes with
Debian Woody.

I like the Mozilla client, but it´s slowlyness is so annoying that would rather
discontinue using it.

Reproducible: Always

Steps to Reproduce:
see Details above
Actual Results:  
looking for another client ...

Expected Results:  
IMAP access should be reasonably fast in general

crash information has been sent several times by the automatic agent

Comment 1

[Boris Zbarsky [:bzbarsky]]

> crash information has been sent several times by the automatic agent

Please post the incident IDs in this bug.

Comment 2

[Hans Wilmer]

PS:

This problem has affected miscelanneous versions since at least 1.3x. I was
hoping that eventually fixes had been implemented in 1.5, but the problem persists.

Comment 3

[David :Bienvenu]

Can you attach some actual numbers? For example, how large an attachment are you
downloading, and how long does it take? Also, do you have any special options
configured, like "check all folders for new mail" or the junk mail filters? Do
you have any folders configured for offline use? What IMAP server are you using?
I'm not often on a high speed LAN, but when I am, things are fast.

Comment 4

[David :Bienvenu]

OK, I set up another IMAP server on my local LAN, copied a large message with an
inline image (1.3MB), and opened the message. It took 3 or 4 seconds to download
and display the message. Not screamingly fast, but not painfully slow either.
Are you seeing performance worse than that?

Comment 5

[Hans Wilmer]

1.) Filtering: I´ve defined a number of filters to move incoming mail to
    the appropriate mailboxes. I did not explicitly disable junk filtering,
    but also, I´m not using any of the filtering options to very great extends.
    (see below, too)

2.) Performance is much worse than what you see. To test it, I saved two
    Attachments from a mail with the ,Save all´ option. Mozilla displays the
    message size as 2134kB; the attachments are 567000 and 1020416 bytes when
    saved.
    It took one minute and 40 seconds to download and save these two
    files with the Mozilla client!
    Saving the same attachments from the same mail with Becky doesn´t take
    any really noticeable amount of time. Becky pops up an annoying warning
    that there are large attachments when I tell it to save them, and after
    confirming the popup, there´s some lack of maybe a second or one-and-a-half
    before the file selection dialog comes up. I guess that the attachments are
    being downloaded during the lack, as it doesn´t take more time to actually
    save the attachments to disk then I´d expect just writing them (from
    memory) would take after I confirm the file selection box.
    So it must be somthing special with the Mozilla client that makes things
    slow. Also, the IMAP server is a pretty fast maschine, and at the current
    time (21:06), I´m the last user here who´s using the LAN (but should go
    home:)
    Well, in case you need some more numbers, I can do some more testing
    tomorrow. Just let me know.

3.) Incident IDs:
    Currently, there appears only one, TB22181255Q. It´s only one because I
    switched to 1.5b recently, but shortly after stopped using the mail client.
    More incidents must have been sent from previous versions, but it seems
    that the information has been deleted by removing the older versions of
    Mozilla. Afair, on crashes, there was some ´collector.dll´ or so named
    often.

4.) Folders
    I tried to disable all offline usage.
    ´Show only subscribed folders´ is disabled, as we´ve some shared ones.
    I´ve explicitly subscribed to all folders because not all subfolders are
      displayed in the folder list. Folders deeper than two levels under the
      INBOX are not found --- to display them, I have to create a new subfolder
      so that I can expand the folder view more. That´s somewhat annoying ...

5.) Mail server
    Cyrus IMAP4 v1.5.19 on Debian Woody, installed from the standard Debian
      Package that comes with the distribution
    Exim is used as MTA.


Other of our users are reporting similar slowlyness-problems since the Mozilla
client has been intodruced as the default mail application. Comments on this
tope are that it takes always long to download/open attachments. So this
phenomen appears on some number of maschines and for a number of users. I don´t
think that they make great use of the mail or junk filtering options. Tomorrow I
could check which version of Mozilla they´re using; just let me know it that
would help.

Moreover, Mozilla is quite a slow browser and mail client in general, especially
on somewhat older maschines. It consumes lots of memory and actually knocked out
some of them. Even on my Linux box at home, with 512 MB of RAM in it and about
another 512 MB swap-partition, memory becomes tight when some browser-tabs are
open ... Nevertheless, it´s the first browser that actually works :)

Comment 6

[David :Bienvenu]

I'm grasping at straws a bit here. Do you have a virus checker installed?

I know about the show all folders bug - I need to try to fix that.

Browser tabs do consume a lot of memory, no doubt. I fixed some bugs very
recently in the mail client where we were holding onto more memory than we
should for the databases for various folders - that was fixed yesterday.

Comment 7

[Hans Wilmer]

A virus scanner is installed on the server (amavisd) to scan all incoming and
outgoing mail before any delivery. --- The clients do not have virus scanners
installed.

I´ll download the latest nightly build of Mozilla and try it out.

Comment 8

[Hans Wilmer]

From: [e-mail]
Message-ID: <[e-mail]>
Date: Fri,  1 Aug 2003 17:29:37 -0400
To: Hans Wilmer <[e-mail]>
Subject: Re: mozilla bug 214637: slowlyness/crashes
References: <[e-mail]> <[e-mail]>
<[e-mail]>
In-Reply-To: <[e-mail]>
X-Originating-IP: 63.190.41.185

Quoting Hans Wilmer <[e-mail]>:

> Thanks for the hint! I=B4ve found it and added the ID to the bug report=
 :)

Great!  Could you also cc [e-mail] and ask him to look up the s=
tacks=20
and assign as appropriate?  Tell him I asked him to do it...

-Boris

P.S.  I'm on vacation and without meaningful bugzilla access for the=20
foreseeable future, hence the roundaboutness of this request.

Comment 9

[Hans Wilmer]

[e-mail] wrote:

> Great!  Could you also cc [e-mail] and ask him to look up the stacks
and assign as appropriate?  Tell him I asked him to do it...


No problem, I´ve added him to the cc list.

BTW, the nightly build I downloaded Friday didn´t crash yet, though I expected
it to. It is still slow, but some improvements seem to have been made :)


Regards,
H. Wilmer

Comment 10

[Christopher Aillon (sabbatical, not receiving bugmail)]

Attachment: Incident ID 22181255

Comment 11

[Hans Wilmer]

Hi,

I´ve just sent another incident, TB22542650G. The mail client was quite slow
when I started it this morning. It seemed to be busy with downloading messages,
as such an information was displayed in the status bar. I opened a mail that was
somewhat larger, read it and then deleted it, pressing the DEL key while the
mail was open. Nothing happened, so I closed the mail and clicked on the DELETE
button of the main window to delete it. After I waited some time (about one
minute), Mozilla crashed.


Some other things that came to my mind yesterday evening:

I´ve been testing perdition for some other purpose, and it might be an helpful
tool in debugging, as it allows you to log the communication that occurs between
the client and the server. Opening a folder with about 12k mails in it, it
seemed that Mozilla downloaded a message header, paused, than downloaded the
next one, and so on. Thus, it takes quite long to open folders with many mails
in them. Where does the pause come from?

As we´ve still a Novell Netware 3.2 server running (using the IPX protocol), all
clients have (miscellaneous versions of) the Novell Netware Client installed.
Imho it´s unlikely, but the slowlyness of the Mozilla Mail Client may come from
some obscure conflicts between the protocols. To verify that, I´ll occassionally
make test on a client maschine before the Novell Netware Client is installed
(It´s about impossible to remove that Netware Client when it has once been
installed ...). In case downloading attachments is significantly faster without
the Netware Client, we might have to live with the slowlyness until the Netware
server is being replaced ... I´ll let you know the outcome.


Regards,
H. Wilmer

Comment 12

[Hans Wilmer]

PS: For perdition, see http://www.vergenet.net/linux/perdition/.

Comment 13

[David :Bienvenu]

We download message bodies because of the junk mail controls. If you turn those
off for your imap account, it will be quite a bit faster to open a folder
because we won't download the message bodies.

We have a logging facility in the client described here:

http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap

I'm not sure it will help, however. We download the message headers all at once,
not one at a time.

When you say "mail was open", do you mean you opened the message in a
stand-alone message window, e.g., by double clicking on it?  That might explain
your crash scenario, since that's not a common useage and we may not have other
reports of it.

Comment 14

[Hans Wilmer]

David,

thanks for the hints! I´ve enabled logging and disabled junk control --- we´ll
see what will happen.


The message was opened in a stand-alone window, by doubleclicking on it. I´ve
switched the message view on the lower right of the client-window down so that
the ´embedded´ display is off. I find it just too small to conveniently view the
messages, so I´m always unsing the stand-alone window. At least half, if not
most of our users, do so.

Comment 15

[David :Bienvenu]

you might also try a trunk build from yesterday or today - I sped up downloading
of large messages somewhat.

Comment 16

[Hans Wilmer]

Thanks a lot! I´ve just started the download ... When it´s finished, I´ll try
downloading some large attachment or so.

Things have improved since I turned off the junk controls. On Monday evening,
I´ve had another crash when closing all Mozilla windows, but the reporting agent
didn´t come up. I´ve kept the logfile from the built-in logging facility, but it
doesn´t seem to contain helpful information. I could mail the logfile or part of
it to you, but it is about 19 MB in size (hm, still almost 8 MB when packed).

Comment 17

[Hans Wilmer]

Downloading and displaying messages seems to be slightly faster now. Saving an
attachment (2003-08-13-mozilla-win32-installer-sea.exe, ~12 MB), takes one
minute and three seconds now. That´s quite an improvement, indeed :)

Comment 18

[David :Bienvenu]

Hans, do you think I can mark this fixed? We could open a separate bug for the
crash, if we can get talkback reports showing the crash in the mail code.

Comment 19

[Hans Wilmer]

David,

well, wheather the bug can be marked as fixed now or not, depends on the
understanding of it, I think. The improved mail client now works for me, but
though downloading IMAP messages and attachments has been noticeably improved,
it could still be faster. Other users have reported the improvement, too.

The junk controls are still turned off to improve performance. Another thing is
the SMTP part on sending mails; it takes just too long to transfer larger mails
to the SMTP server (but I don´t know if that is a known issue). Except the crash
on last Monday, there hasn´t been another one.

Maybe we should mark this bug as fixed and eventually open new ones concerning
SMTP and junk-controls, thereby keeping in mind that further improvements in
IMAP speed would be welcome :)


GH

Comment 20

[David :Bienvenu]

Hans, that sounds like a good idea.  It's better to have separate bugs for
separate problems, and SMTP and IMAP are very separate :-) The junk mail
controls are always going to slow things down because they cause us to download
the whole message body