Thunderbird 2.0.0.0 uses 100% CPU when fetching IMAP emails with junk mail controls on - huge .msf file

RESOLVED WORKSFORME

Status

MailNews Core
Database
--
major
RESOLVED WORKSFORME
11 years ago
8 years ago

People

(Reporter: James, Unassigned)

Tracking

({perf, qawanted, regression})

1.8 Branch
x86
Linux
perf, qawanted, regression

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [closeme 2009-12-01][needs retesting by reporter])

(Reporter)

Description

11 years ago
User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.8.1.3) Gecko/20070324 Firefox/2.0.0.3
Build Identifier: version 2.0.0.0 (20070418)

Whenever I receive new emails, Thunderbird 2 goes to 100% CPU, then after finishing receiving the emails, with them now in the inbox, it returns back to normal CPU utilization. Thunderbird 1.5 did not have this problem.
 

Reproducible: Always

Steps to Reproduce:
1.
2.
3.

Comment 1

11 years ago
bug 320515 comment 5 could be of help
(Reporter)

Comment 2

10 years ago
I am currently using Thunderbird 2.0.0.6
This is still a problem. The performance compared to eudora or another email client using imap is considerable.
Performance is improved is one switches off the junk mail filters.
I think performance would be largely improved if the junk mail filters scanned emails before running the filters.

Comment 3

10 years ago
James, what happens if you test out 
* bug 358595 comment 10?  
* bug 358595 comment 21?


Also, from bug 358595 comment 5 "To check if a message is junk, we download the message and tokenize the text of it. ... suggest adding filters to mark messages that you know aren't junk (e.g., your mailinglist postings) as non-junk - that will prevent us from downloading the message bodies to analzye them for junk status."

David ...

(In reply to comment #0)
> Whenever I receive new emails, Thunderbird 2 goes to 100% CPU, ... Thunderbird 1.5 did not have this problem.
(In reply to comment #2)
> Thunderbird 2.0.0.6
> Performance is improved is one switches off the junk mail filters.
Version: unspecified → 2.0

Comment 4

10 years ago
hmm, reporter gone?
Assignee: mscott → nobody
Whiteboard: closeme 2008-03-01
Resolving closeme bugs that did not have any useful information provided since the last comment. -> incomplete.

Please feel free to comment if the issue still occurs in the latest supported version 2.0.0.12 or trunk nightlies.

Filter on ENDEAVOURLANDSATNIGHT123
Status: UNCONFIRMED → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → INCOMPLETE
(Reporter)

Comment 6

10 years ago
I tried all the suggested "fixes" but nothing helped.
I use IMAP, and side by side, eudora or kmail makes the email appear almost instantly, whereas thunderbird takes about 60 seconds per 10 messages.
Status: RESOLVED → UNCONFIRMED
Resolution: INCOMPLETE → ---
(Reporter)

Comment 7

10 years ago
I have done some diagnosis on this now.
It is spending this 100% CPU doing stuff like this.
-1224812848[805c548]: add word: regions, (count=38)
-1224812848[805c548]: adding word to tokenizer: regions, (len=8) (count=38)
-1224812848[805c548]: add word: efcf7fa8 (count=1)
-1224812848[805c548]: adding word to tokenizer: efcf7fa8 (len=8) (count=1)
-1224812848[805c548]: add word: date:10/04/07 22:30 (count=1)
-1224812848[805c548]: adding word to tokenizer: date:10/04/07 22:30 (len=19) (count=1)
-1224812848[805c548]: add word: holders') (count=2)
-1224812848[805c548]: adding word to tokenizer: holders') (len=9) (count=2)
-1224812848[805c548]: add word: ptwrite, (count=4)
-1224812848[805c548]: adding word to tokenizer: ptwrite, (len=8) (count=4)


This was with log all:5
I have turned off my junk mail filter and it still does this. What is happening?

Updated

10 years ago
Keywords: perf, qawanted, regression
Whiteboard: closeme 2008-03-01

Comment 8

10 years ago
what is your mail check interval set to?
Component: General → MailNews: Filters
Product: Thunderbird → Core
QA Contact: general → filters
Version: 2.0 → 1.8 Branch
(Assignee)

Updated

10 years ago
Product: Core → MailNews Core

Comment 9

9 years ago
hello James?  question in comment 8

also see Bug 358595
Whiteboard: closeme 2009-01-10
(Reporter)

Comment 10

9 years ago
Check interval is set to never. I.e. I have to click on the "get mail" or the inbox to update incoming emails.
I have looked at Bug 358595, and have switched off the preview pain.
I will have to wait and see if this helps.
I already have the junk mailer switched off.

Comment 11

9 years ago
James, how does it look now?

Updated

9 years ago
Whiteboard: closeme 2009-01-10 → closeme 2009-02-10

Comment 12

9 years ago
Closing the preview pain improves things a bit, but it still takes a lot of time.
It still hits 100% CPU just to collect 100 messages.
I would have though that an email program would be more disk intensive than CPU intensive.
The cyrus imap email server sits at about 0% during this time.
Problem still present with thunderbird 2.0.0.19
Other linux imap clients do not have this 100% CPU problem.

Comment 13

9 years ago
I did some tests.
If email appears in a folder and does not get filtered to move it to other folders, the emails appear at about 1000 every 3 seconds. This matches other linux email applications.
If the email appears in the inbox and gets filtered to be moved to other imap folders, it handles about 10 every 60 seconds.
I suppose a possible solution to this problem is to use server side filters.
Does thunderbird have a feature whereby it can transfer the filters to the cyrus sieve server side filters?

Comment 14

9 years ago
I have found something useful.
I now have a mailbox that has 125 mail messages waiting.
It jumps to message 31 very quickly, but then sits at 31 on 100% CPU.
With the IMAP:5 logging, there is in fact no IMAP chatting at all while at 100% CPU.
This obviously points to some other system apart from IMAP causing the 100% CPU problem.
What should I try next?

Comment 15

9 years ago
(In reply to comment #14)
> I have found something useful.
> I now have a mailbox that has 125 mail messages waiting.
> It jumps to message 31 very quickly, but then sits at 31 on 100% CPU.
> With the IMAP:5 logging, there is in fact no IMAP chatting at all while at 100%
> CPU.

- is this with junk processing turned off?
- how big is your training.dat file?
  http://kb.mozillazine.org/Profile_folder_-_Thunderbird
- do you have other filters in thunderbird?


(personally, I filter most if my junk and other stuff to local folder)
Whiteboard: closeme 2009-02-10

Comment 16

9 years ago
Using strace, while on 100% CPU, it is spending all of its time reading files
in the local .thunderbird folder without any IMAP activity at all.

Comment 17

9 years ago
junk processing is turned off.
I have filters enabled that simply look at headers in incoming messages and move the messages to different folders. I use it for mailing lists.

Further analysis has shown that the 100% CPU is taken reading .msf files.
I do not know what the .msf files are, but as an example, it is taking over 35 seconds (I killed it after 35 seconds or 100% CPU)
Most .msf files take 1 second or less, but one is taking over 35 seconds.
The files in the .thunderbird directory seem to be paired.
e.g. "foldername" is paired with "foldername.msf"
In this example, the foldername is 1187778 bytes long, and the foldername.msf is 45648659 bytes long.
It appears to me that foldername stores the actual emails and foldername.msf stores some sort of index. So, why is the index file so much larger than the file it is indexing? Is there some way to reduce the size of the .msf file?

Comment 18

9 years ago
Further analysis now shows me that when one "compacts" a mail folder, the "foldername" size does decrease as emails are expunged, but the associated .msf file does not get compacted, so that over time the .msf file seems to get very large.
For example, after I exited thunderbird, deleted the previous "foldername.msf" that was taking more than 35 seconds to process, and restarted thunderbird, the size of the .msf file has changed from 45648659 to 259888!!!
The folder now takes less than a second to process.
Suggestion: Redo the .msf file when a folder is compacted.

Comment 19

9 years ago
When I did a global delete of all .msf file, the system now processes the 125 emails in less than a second.
It also took only about 5 seconds to recreate all the .msf files.

Comment 20

9 years ago
Method that I used to diag the fault.
strace -o thund-1 -ff -ttt /usr/bin/thunderbird
at the same time run "top" with "shift-h" to find out which thread is at 100%
Then do the operations in Thunderbird to reproduce the problem.
Then kill thunderbird and examine the strace of the particular thread found in "top" to determine what it was doing when at 100%

Comment 21

9 years ago
James, was the folder with the giant .msf file an imap folder or a local folder?

Comment 22

9 years ago
It was an imap folder.
James - could you try Thunderbird 3.0b2 and tell us if things are similar with that version ?

Updated

9 years ago
Whiteboard: dupeme

Updated

8 years ago
Severity: normal → major
Component: Filters → Database
QA Contact: filters → database
Summary: Thunderbird 2.0.0.0 uses 100% CPU when fetching IMAP emails with junk mail controls on. → Thunderbird 2.0.0.0 uses 100% CPU when fetching IMAP emails with junk mail controls on - huge .msf file

Comment 24

8 years ago
please check version 3 when it's released
Whiteboard: dupeme → [closeme 2009-12-01][needs retesting by reporter]

Comment 25

8 years ago
I don't know of any junk processing issues, so I'm going to close this WFM as of version 3 which is about a week away.  When you begin to use version 3:
* if you see same problem please reopen the bug
* if you do NOT see the problem, please comment so we can in fact verify that the problem is gone for you

v3 is currently being tested http://www.mozillamessaging.com/en-US/thunderbird/early_releases/
Status: UNCONFIRMED → RESOLVED
Last Resolved: 10 years ago8 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.