Closed Bug 579520 Opened 14 years ago Closed 13 years ago

IMAP is dog slow downloading messages with indexing off

Categories

(Thunderbird :: Folder and Message Lists, defect)

Product:
Thunderbird
Component:
Folder and Message Lists
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: a_geek, Unassigned)

Details

(Keywords: perf) 

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.10) Gecko/20100504 Firefox/3.5.10
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.10) Gecko/20100512 Lightning/1.0b1 Thunderbird/3.0.5

For testing purposes, I have an IMAP server (Dovecot 1.2.12) on my computer, which I access through Thunderbird. The computer has an otherwise really idle AMD 945 CPU, and gobs of free memory. So I put some 6300 messages into the inbox, which I wanted to have sorted by TB, according to a very simple subject filter: If the subject contains a certain fixed string, move the message to one other folder in that same mail account. This took some 7:45 minutes of CPU time (~9 minutes of wall clock time?) on said computer while one of the four CPU cores was pegged at 100% usage. TB would almost not react to any user input, nor failed to update the folder display to show the new message count until I clicked around quite a bit.

Folks, this is just MUCH TOO SLOW.

Also, TB was _very_ sluggish thereafter until I killed it (just choosing "exit" from the menu was insufficient), and restarted it.

Reproducible: Always

Steps to Reproduce:
1. Have a non-insignificant number of messages in an IMAP mailbox.
2. Start TB and access that mailbox.
3. See how TB hogs the cpu and becomes almost unusable, due to sluggish reactions.
Actual Results:  
TB becomes almost unusable and needs to be restarted to get sane reaction times again. Unusable means that it takes something like 20 seconds to open a short 500 bytes text message.

Expected Results:  
TB should tear through these few messages with ease, and in a blink of an eye.
If it's about the moving of the messages (which implies a copy+delete), then it should be fixed in Thunderbird 3.1 ; see bug 530098. Please upgrade if you can.
I've upgraded to 3.1 (build ID: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.4) Gecko/20100608 Lightning/1.0b2 Thunderbird/3.1) and downloaded new messages from the same server. This time, I had 130 new messages in my inbox. It took me some 30+ seconds to download them (from localhost, mind you), and some 10-20 seconds more to have them filtered. I'd say that this isn't exactly an improvement. As before, the computer wasn't doing anything noticable else, and, as before, TB used 21 CPU seconds so far, not doing much else than this one thing of downloading and filtering those 130 messages.
understand your concerns, but you will want to decide which issue this bug is to cover - b. the "move" message issue or c. the slow download.

but for now
a) is this a fresh install of v3.x? or are you coming from v2?
b) if by "move" message you mean drag then your issue is a different, existing bug
c) regarding slow download it doesn't sound like a known thunderbird issue. do you run a firewall?

(also, it's helpful if you briefly state your exact actions as numbered steps - parsing more words is more difficult)
Component: General → Folder and Message Lists
Keywords: perf
QA Contact: general → folders-message-lists
I don't remember correctly whether I freshly installed, or whether I upgraded from 2.x, but if so, I've done that in the distant past because I was one of the first users of 3.0.

"Move" means: TB downloads the messages, and the configured filter does its work. My attributions of TB's usage of CPU time include only one human action, and that is pressing "get all messages" for this IMAP account, and then confirming the password. After that, I only watch the clock, top, etc., until TB is finished (marked by updating the message count in the two folders of that account).

Yes, I do run a firewall, but that doesn't filter connections on localhost, where the IMAP server is located.
in my mind this doesn't add up to being a bug. for example you say you were an early adopter of 3.0 (8 months ago or more?) and yet you only just filed this bug against 3.0, which isn't a great indicator that the problem is thunderbird. it seems to me more likely that something in your environment has changed that affects thunderbird.

you'll want to look at your environment again and perhaps also ..
* you of course tested in safe mode, but if not, then https://support.mozillamessaging.com/en-US/kb/Safe+Mode
* if your thunderbird memory usage is high go through https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems
* try disabling certain things in thunderbird, including filters (you don't indicate you did so)
* how big is your global-messages-db.sqlite file in profile directory?
* how big are those messages that are being downloaded?
Summary: IMAP is dog slow → IMAP is dog slow downloading messages
This bug was fixed earlier in the summer, for the 3.1.3 or 3.1.4 release.
But.. it's back.. 3.1.5 is once again DOG SLOW with IMAP.
Reverting back to 3.1.4 speeds it all up again.

I cannot find the original bug thread where the original fix was posted/discussed -- it was something about fixing thunderbird to not open a new connection for each header/message downloaded or something like that.

Ugh.
a_geek, please reply to comment 5. TIA


(In reply to comment #6)
> This bug was fixed earlier in the summer, for the 3.1.3 or 3.1.4 release.
> But.. it's back.. 3.1.5 is once again DOG SLOW with IMAP.
> Reverting back to 3.1.4 speeds it all up again.
> 
> I cannot find the original bug thread where the original fix was
> posted/discussed -- it was something about fixing thunderbird to not open a new
> connection for each header/message downloaded or something like that.

you'd want to look at the list of bugs fixed mentioned in the release notes.
Whiteboard: closeme 2010-11-15
I now started TB in safe mode. Observations so far:

* TB (Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.11) Gecko/20101013 Thunderbird/3.1.5) now uses about half as much memory. With add-ons: 560772, without: 248440, as displayed by 'ps auwwx|grep thund'. Memory isn't my primary concern, as I have enough of it. My addons list for TB is short: Enigmail, CompactHeader, Extra Folder Columns, and Lightning.

* When not running in safe-mode, TB usually starts normally, but suddenly, out of the blue (ie, TB sits there, idle, while I work on a different virtual desktop on other things), TB suddenly grabs 100% of one CPU core for minutes, then falls down to normal again. This then happens frequently. When it happens, TB is nigh unusable - eg. switching to the desktop where TB has its window and trying to see the window takes, say, 20, 30 seconds before the window is being mapped. Typing lags behind up to 20, 30 seconds as well, but not always.

* While running in safe-mode, I experienced similar behaviour as well. I don't know whether I have triggered that behaviour, but it occurred while I was scrolling through a mailbox containing some 6200 messages (which is not very much, imho).

The bulk of my email that I actually process with TB, is written in Chinese. Maybe encoding problems and/or incorrect handling of multibyte texts may be influential to this problem. But of course, I get spam in all languages and broken encodings, too, so there's even more room for failure in that area.

I have been running TB in safe-mode for about 19 minutes.
(In reply to comment #6)
> This bug was fixed earlier in the summer, for the 3.1.3 or 3.1.4 release.
> But.. it's back.. 3.1.5 is once again DOG SLOW with IMAP.
> Reverting back to 3.1.4 speeds it all up again.


TB 3.1.6 is out today, and iMAP is quick again (for me) with the new release.
Thanks.
(In reply to comment #8)
> I now started TB in safe mode. Observations so far:
> 
> * TB (Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.11) Gecko/20101013
> Thunderbird/3.1.5) now uses about half as much memory. With add-ons: 560772,
> without: 248440, as displayed by 'ps auwwx|grep thund'. Memory isn't my primary
> concern, as I have enough of it. My addons list for TB is short: Enigmail,
> CompactHeader, Extra Folder Columns, and Lightning.

your findings are consistent with the fact that calendar uses lots of memory. 
(note - the memory question is asked, because high memory usage by itself can be symptomatic of problems that cause performance issues that won't be helped by having enough memory)

> * When not running in safe-mode, TB usually starts normally, but suddenly, out
> of the blue (ie, TB sits there, idle, while I work on a different virtual
> desktop on other things), TB suddenly grabs 100% of one CPU core for minutes,
> then falls down to normal again. This then happens frequently. When it happens,
> TB is nigh unusable - eg. switching to the desktop where TB has its window and
> trying to see the window takes, say, 20, 30 seconds before the window is being
> mapped. Typing lags behind up to 20, 30 seconds as well, but not always.

you don't indicate you did all of https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems ... so you need to do #8, 9, 11, 12. and also try disabling compose auto-safe draft in options > composition > general
also, do you have sound enabled for new messages?  if you do, please try disabling. (ref bug 367431)
Mark, something external is affecting your performance - 3.1.6 didn't have any changes from 3.1.5 that would affect performance.  Virus checkers/anti-spam programs are common culprits.
Wayne: I'm looking through the points you make. I just missed the wiki page, and am going through it right now. I have 8gigs of RAM in the box, so if TB uses 1/4 or 1/2 gig of that, I currently don't care. To the performance figures:

1. Indexing was already switched off.
2. I don't know whether I have sound on or off, but usually, I don't have sound on my computer, so I don't notice.
3. I have gkrellm running, too, and didn't notice extraordinary network traffic, but will check with the newsgroups disabled. In fact, I have four news servers configured, and message retention configured to be between 15.000 and 50.000  messages.
4. I'm not sure how much memory TB should normally consume, but 250MB is ok for me.
5. I use text-only mails, but still don't know how to really disable it in the composing window. That notwithstanding, people are - unfortunately - sending HTML messages my way.
6. I have the junk filter enabled.
7. I didn't notice "RAM breathing" (#541001).
8. I've now checked my folder sizes, and the largest folder was some 220MB or so.
on rereading comment 8 and comment 3 I was reminded that you seem to be shifting between different problems, which are now up to at least 3 problems (all of this reproduce in safe mode). That's causing confusion and making things difficult to understand in this bug.  It may be best to send your issue to the support forum @ http://getsatisfaction.com/mozilla_messaging/topics/new

but to summarize the issues you stated so far:
a) download speed 
b) filter speed
c1) "switching to the desktop where TB has its window and trying to see the window takes, say, 20, 30 seconds before the window is being mapped. 
c2) Typing lags behind up to 20, 30 seconds as well, but not always."
d) "occurred while I was scrolling through a mailbox containing some 6200 messages"

c1) switching doesn't sound like a thunderbird problem, unless c2) is a thunderbird problem and the two are related.

d) need more information - what do you mean by scrolling - using arrow keys to step through messages, our using scroll bar with mouse?

A possibility is some background activity on your machine is causing delays in Thunderbird. So, some things to test:

* idle your server and all other software on the machine, and try the things in thunderbird (except downloading) that have been problems - is problem gone?
* if not gone, make thunderbird offline (file > offline > work offline) and repeat - is problem gone?
Summary: IMAP is dog slow downloading messages → IMAP is dog slow downloading messages with indexing off
and regarding sound - it doesn't help that you have sound off on your machine. what matters is whether you have it enabled in thunderbird. what do you have set for
  preferences > general > play sound
Wayne: I have the sound button unchecked in Preferences -> General.
Wrt. speed: It's only TB which is slow, all other programs (FF, Emacs, a variety of xterms, etc.) all are running at normal speed at the same time when TB is slow. I can also see it in 'top' as being the single one offender. I have a 945 machine w/ 8GB of RAM and 2x1TB SATA server disks as a RAID1. I don't currently know the cpu speed per specs, but dmesg says "Total of 4 processors activated (24106.66 BogoMIPS).". IOW, the machine should easily be capable of running TB with acceptable performance. When TB is slow, one and only one core is pegged at 100%, while all other cores idle along at 0-5%. I continuously have gkrellm running which shows the disks and network with negligible activity at such times. By scrolling, I mean trying the mouse wheel or the cursor keys. I usually use the mouse wheel, or try to pull the slider, and I'm really mostly concerned with interactive usage which becomes slow, and can well ignore the other speed issues I'm having. I'm quite certain that no background task is slowing TB down.
can you reproduce this with thunderbird on a different machine?
please attach a protocol log file to the bug for 
  imap:5,autosync:5,timestamp
per https://wiki.mozilla.org/MailNews:Logging
Whiteboard: closeme 2010-11-15
I'm currently not able to try a different machine, simply because I don't have a suitable machine right now. But FWIW, the next machine, due in a few weeks, will most likely run the same software (ie, Debian Testing/Squeeze, i386).
my instincts point the finger at lightning.  if you confirm the problem is far less severe with lightning disabled, then the bug should move to calendar for diagnosis.

a geek, what do you say?? :)
(In reply to comment #17)
> can you reproduce this with thunderbird on a different machine?
> please attach a protocol log file to the bug for 
>   imap:5,autosync:5,timestamp
> per https://wiki.mozilla.org/MailNews:Logging

or even same machine.

> a geek, what do you say?? :)
Whiteboard: [closeme 2011-02-21][need a geek update]
slow mousewheel scroll is bug 356201. 

Sounds like the original problem reported in comment 0 with version 3.0, is resolved
Status: UNCONFIRMED → RESOLVED
Closed: 13 years ago
Resolution: --- → WORKSFORME
Whiteboard: [closeme 2011-02-21][need a geek update] → [closeme 2011-02-21][test in safe mode]
(In reply to comment #21)
> slow mousewheel scroll is bug 356201. 
> 
> Sounds like the original problem reported in comment 0 with version 3.0, is
> resolved

... the original problem minus calendar and whatever other addons are installed - whose issues should be reported seperately.
Whiteboard: [closeme 2011-02-21][test in safe mode]
Still a big problem in Thunderbird 17.0.

But I have noticed something that I hadn't discovered before.

While Thunderbird is slowly scanning my 3000 new messages of the day,
at about 10/second, If I switch away from the Inbox folder to another
folder on the same account, it all suddenly all completes at lightening speed.

So perhaps some kind of contention with the GUI on the Inbox folder?
You need to log in before you can comment on or make changes to this bug.