548784 - High memory consumption when Global Search and Indexing is turned on

Status: RESOLVED INCOMPLETE

(MailNews Core :: Backend, defect)

Description

[xephor]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2) Gecko/20100115 Firefox/3.6 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.8) Gecko/20100216 Lightning/1.0b1 Thunderbird/3.0.2

Thunderbirds memory usage increases after start up and it causes a high CPU load (50%) for approx. 5 minutes - during this phase memory consumption is as high as 600 MB.
After around 5 minutes the CPU load stops (0-2%) and the memory usage stays constantly around 468 MB of memory.
After some research in the web and the bugzilla database I followed the procedure described at: https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems
Steps 1 through 8 did not resolve the problem, step 9 (disabling global indexing) did
=> Memory consumption is now around 76 MB

As the other bugs I found (e.g. 541001, 337837) seam to have other reasons I decided to file a bug.

Reproducible: Always

Steps to Reproduce:
1. Start Thunderbird
2. Activate Global Indexing
3. Restart Thunderbird
Actual Results:  
CPU load > 50% for approx. 5min
Memory consumption above 468 MB


Expected Results:  
Memory consumption below 100 MB

Comment 1

[Wayne Mery (:wsmwk)]

with global indexing enabled, if after 5 minutes the memory doesn't drop to less than 200MB (down from ~468MB) then please attach a msgdb:5 log file for that time period.

Comment 2

[xephor]

Attachment: ~6 minutes of logging after startup with global indexing turned on

Thank you for your reply! I have never done such a logging before, so I hope the attached log file is what is needed.
I stopped Thunderbird after approx. 6 minutes with a memory consumption of ~800MB.

Comment 3

[Wayne Mery (:wsmwk)]

Thanks xephor.  Did you get here via getsatisfaction topic?

-> backend until we know more

Comment 4

[xephor]

The short answer is yes.

The long answer:
- I have read similar topics at getsatisfaction
- That lead me to bugzilla
- Then I found the mentioned memory article in the wiki
- I followed the steps and finally from there I got back to bugzilla and opened this bug

Comment 5

[xephor]

Problem still occurs with latest version 3.0.3 of Thunderbird (Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.8) Gecko/20100227 Lightning/1.0b1) Thunderbird/3.0.3

Comment 6

[Wayne Mery (:wsmwk)]

xephor, are your accounts imap or pop?  
can you check for similarities in the comments of the bugs below and post any findings here?  
(it's interesting that this maxes out at 19 open dbs)

imap xref  Bug 526568 -  Consumes too much memory, nearly 2 GB with gloda indexing on, 800MB with gloda indexing off
pop xref  Bug 542234 -  Thunderbird 3 - memory increases about 12MB on each check for new mail

Comment 7

[xephor]

My main account is imap, but I have one little pop3 as well.

I checked for similarities with Bug 526568, but I find them to be not really matching. Especially I looked at the following points:

* I do have approx. 30 folders in it with approx 1200 messages in them.

* My global-messages-db.sqlite file in your profile was over 2GB of size. (I have deleted it in the mean time - now it as 343 MB)

* Setting mail.check_all_imap_folders_for_new to false (and disabling almost all "Check folder for new messages") did not change the behaviour.

Bug 542234 seems not to be matching, because the behaviour here seems not to be related to each pop check for new mail - memory consumption changes constantly and is generally increasing. Behaviour still occurs if I deactivate the mail checking for the pop account.

Two additional notes:
* Some time ago I copied approx. 15 folders from my old IMAP account to my new IMAP account
* Sometimes I have to kill Thunderbird, because it will not stop running when I quit it

Comment 8

[David :Bienvenu]

are you using Enigmail, or other extensions?

Comment 9

[Andrew Sutherland [:asuth] (he/him)]

This sounds like a combination of:
https://bugzilla.mozilla.org/show_bug.cgi?id=543737 and
https://bugzilla.mozilla.org/show_bug.cgi?id=530098

Comment 10

[xephor]

Yes, I am using Enigmail and other extensions, too

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to comment #10)
> Yes, I am using Enigmail and other extensions, too

comment 0's "https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems
Steps 1 through 8 did not resolve the problem" should mean that Enigmail and other extensions don't have an impact, or at least not a major impact.

Comment 12

[David :Bienvenu]

If this still happens in safe mode, then it's not enigmail, but as I understand it, enigmail can cause both these symptoms.

Comment 13

[xephor]

I do not know, if this is related - but I noticed today, that Thunderbird does not always show me the correct message. 
I wanted to view a message I received to day - if I click on it in the upper message list window, it shows me another older message in the message detail window below.
Only thing that helped was to move the message to another folder - viewing it there, worked then.

Comment 14

[Wayne Mery (:wsmwk)]

xephor, how much is your "resting state" memory usage again? And can you check again this *does* happen in safe mode?

Comment 15

[xephor]

Sorry that I was not able to test this earlier.

Short answer:
Wayne, this does not happen (any more) in safe mode.

Long answer:
Wayne, I did the following to answer you question:

- Started thunderbird.exe -safe-mode
- Memory consumption according to Win Task Manager: ~56200K, CPU: 0%
- Extras -> Preferences - Advanced -> General: Turned on global search an indexing
- Memory and CPU consumption stay the same
- Quit Thunderbird
- Started thunderbird.exe -safe-mode
- Memory: ~72.000K, CPU: ~15%
- After 3min: Mem ~87.000 K, CPU: 0%
- After 5min still the same as after 3min
- I started a search in all accounts (Search window in upper right corner)
- After 6min: Mem: ~348.500K, CPU: ~40%
- After 7min: Mem: ~548.000K, CPU: 0%, search results are shown
- After closing search results: Mem: ~62.800K, CPU: 0%

In safe mode everything works fine now
May be there have been changes since version 3.0.2 to 3.0.4 (which I am using now)?

I will use Thunderbird in normal mode for a while with global search and indexing turned on an tell you about the behaviour.

Comment 16

[Ludovic Hirlimann [:Usul]]

If everything works fine in safe-mode than the issue is within an extension.

Comment 17

[xephor]

Current Thunderbird version I am using is: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.9) Gecko/20100317 Lightning/1.0b1 Thunderbird/3.0.4

Comment 18

[Wayne Mery (:wsmwk)]

xephor, what is your full list of extensions?

recap as I understand it...
comment 0: ~500MB-600MB for about 5 minutes then settles down to <100MB
comment 15: 
a) startup 1 "normal" - safe mode & indexing is off
b) startup 2 normal until search - safe mode & indexing is on, and closing search results then drops to 62MB!

I don't see how comment 0 is substantially different from comment 15. Nor do I see how safe mode affected anything. Or perhaps I don't correctly follow comment 15. Lastly, I think searching shouldn't suck up 450MB and the drop back to normal.  can you reproduce and capture that time span in a msgdb log.

(fwiw, someone in getsatisfaction recently claimed MoreFuctionsForAddressBook caused memory problem)

Comment 19

[xephor]

Hello everybody,

sorry for the long delay. Unfortunately my hard disk crashed and I had to move my stuff to a new one :(

Now, with a freshly installed Thunderbird, I am not able to reproduce this behavior any more...sorry.

But I want to thank you all for the help and support - I really appreciate your efforts!

Last but not least to answer Wayne Merys questions:
- I believe my extension were: British Dict., German Dict., US Dict., Enigmail, Lightning, MinimizeToTrayPlus

- The difference between comment 0 and comment 15 is:
  * that in comment 15 the high memory only appeared with an opened search window (and goes away when closing the search)
  * in comment 0 it appeared when opening Thunderbird with the message list (no search window open) and just stayed that way no matter which actions were performed

Comment 20

[Wayne Mery (:wsmwk)]

flips the incomplete/worksforme coin, and it comes up ...
=> incomplete  (per random neurons)