557001 - Thunderbird 3 uses lots of memory with indexing enabled

Status: RESOLVED WORKSFORME

(MailNews Core :: Backend, defect)

Description

[Dimitri Papadopoulos]

User-Agent:       Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; InfoPath.2)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.0; fr; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4

I've followed instructions here to diagnose the problem:
https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems

In step 9 disabling global indexing does solve the problem (at least partly). Memory usage goes down from 500 MB to 35 MB.
Re-enabling global indexing and restarting Thunderbird brings back the problem: Thunderbird uses 500 MB or even 600 MB memory and makes teh whole system unusable.
Additionally Thunderbird needs 10 or 20 seconds to stop with global indexing enabled.

I'm using Thunderbird to access a Gmail account through IMAP.

Reproducible: Always

Steps to Reproduce:
1. Set up Thunderbird 3 to access a Gmail account with IMAP.
2. Start Thunderbird.
3. See memory usage rise to 500 MB and more.
Actual Results:  
Uses 500 MB of memory.

Expected Results:  
Should use 5 MB of memory like Windows Mail.

Comment 1

[Wayne Mery (:wsmwk)]

Dimitri, with indexing turned on, can you attach a msgdb log per https://wiki.mozilla.org/MailNews:Logging
Thanks

Comment 2

[Dimitri Papadopoulos]

Attachment: msgdb log

Output of:
set NSPR_LOG_MODULES=MSGDB:5
set NSPR_LOG_FILE=d:\msgdb.log
"C:\Program Files\Mozilla Thunderbird\thunderbird.exe"

I started Thunderbird with indexing turned on and memory usage rose steadily to reach in less than a minute a state where memory usage oscillates between 300 MB and 600 MB. At that point I stopped Thunderbird since it almost froze the PC.

Hope it helps. Would be happy to provide additional information.

Comment 3

[Wayne Mery (:wsmwk)]

only 5 open dbs

Comment 4

[David :Bienvenu]

Hi Dimitri, thx for the log. It shows that the memory is not used by .msf files, unless you have some giant .msf files. I would suspect it's more likely gloda indexing using memory. If you turn off gloda indexing, is memory usage more normal?

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

The 2038 open headers implies gloda got busy.  It occurs to me that the JS mime emitter might be producing ridiculously large amounts of string garbage...

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

In fact, I'm going to look into that right now...

Comment 7

[Dimitri Papadopoulos]

Turning off global indexing does help: memory usage drops to 50 MB. That's still more than the memory usage of Windows Mail on the very same account (5 MB if I recall correctly) but it's at least it's usable.

The .msf files are probably "large" - this the GMail account of a person who almost never deletes mails...

Comment 8

[Dimitri Papadopoulos]

Just to make sure... I understand turning off gloda indexing means unchecking:
Tools | Options | Advanced | General | Advanced configuration | Enable global search and indexer

Comment 9

[Dimitri Papadopoulos]

Attachment: dir /S imap.gmail.com*

Please find attached the output of:
cd AppData\Roaming\Thunderbird\Profiles\7u7iiyr2.default\ImapMail
dir /S imap.gmail.com*

This will show the size of .msf files.

Comment 10

[Andrew Sutherland [:asuth] (he/him)]

Investigation suggests that the JS mime emitter's use of temporary strings is unlikely to be directly responsible for the memory bloat unless it is doing it by fragmenting memory quite horribly.  (This is conceivable although hopefully not likely.  Because of JS' support for dependent strings the += ends up with the string we are appending to slowly growing through reallocation while the RHS strings end up as garbage.  The memory allocator would have to be pretty naive to destroy memory for that...)  On my machine, at least, the most string data I saw accumulated between GCs on a reindex of a folder with 4600 messages was 1.3 megs with the JS string object count at ~45k with estimated cost at 720k of memory on a 64-bit system.

I think we can probably nail this one without too much additional effort by just sampling what is causing the VM growth.

Comment 11

[Dimitri Papadopoulos]

I can reproduce the problem with the same Gmail account on a 64-bit Linux workstation, using the version of Thunderbird 3.0.4 bundled with Fedora 12.
Of course memory usage is not a problem on this machine since it has 4 GB of RAM.

Example of "top" output:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
27889 myoung    20   0  855m 149m  28m S 44.8  3.8   1:20.76 thunderbird-bin

Comment 12

[Wayne Mery (:wsmwk)]

existing bug reports for mime+memory https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=anywordssubstr;value2-0-1=mlk%20perf;field0-0-0=short_desc;value2-0-0=slow%20memory%20leak;type1-0-1=substring;field2-0-0=short_desc;value1-0-1=mim;resolution=---;classification=Client%20Software;classification=Components;type2-0-0=anywordssubstr;query_format=advanced;type2-0-1=anywordssubstr;value1-0-0=mime;type0-0-0=nowords;value0-0-0=count%20counts;field1-0-0=short_desc;product=MailNews%20Core;product=Thunderbird;field2-0-1=keywords;field1-0-1=component

Comment 13

[Dimitri Papadopoulos]

I have checked this problem has gone away using Thunderbird 3.1.4 on Windows.

On Windows Thunderbird now needs ~ 45 MB RAM. Note that the options dialog has changed. To disable Windows indexing I had unchecked:
Tools | Options | Advanced | General | Advanced configuration | Enable global
search and indexer
To enable it again I had to check:
Options | General | Let Windows search messages

On Linux Thunderbird still uses a little bit more memory but as I said that's not a problem since this machine has 4 GB of of RAM:
21577 myoung    20   0  826m  97m  30m S  1.2  2.5   0:10.01 thunderbird-bin

Comment 14

[Wayne Mery (:wsmwk)]

(In reply to comment #13)
> I have checked this problem has gone away using Thunderbird 3.1.4 on Windows.
> 
> On Windows Thunderbird now needs ~ 45 MB RAM. Note that the options dialog has
> changed. To disable Windows indexing I had unchecked:
> [1] Tools | Options | Advanced | General | Advanced configuration | Enable global
> search and indexer
> To enable it again I had to check:
> [2] Options | General | Let Windows search messages

Dimitri, [1] and [2] are not for the same functions/search type. So if I understand you correctly, then you are not testing with indexing off AND on

Comment 15

[Dimitri Papadopoulos]

Oops... I may indeed be testing different options.
How do to enable/disable global search and indexer in Thunderbird 3?
Can you point me to documentation?

Comment 16

[Ludovic Hirlimann [:Usul]]

Comment 8 is correct. but you stll have a database so you need to remove global-messages-db.sqlite from your profile to effectively disable it's use.

Comment 17

[Dimitri Papadopoulos]

Just to make sure...

Comment 8 describes how to enable/disable gloda indexing in Thunderbird 2.

Comment 14 explains I might have not enabled gloda indexing properly in Thunderbird 3 so I might be comparing Thunderbird 2 and 3 with different parameters.

How to enable/disable gloada indexing in Thunderbird 3?

Comment 18

[Ludovic Hirlimann [:Usul]]

(In reply to comment #17)
> Just to make sure...
> 
> Comment 8 describes how to enable/disable gloda indexing in Thunderbird 2.

Gloda didn't exist in v2.

> Comment 14 explains I might have not enabled gloda indexing properly in
> Thunderbird 3 so I might be comparing Thunderbird 2 and 3 with different
> parameters.
> 
> How to enable/disable gloada indexing in Thunderbird 3?


coment 8+ 16 to disable gloda.

Comment 19

[Dimitri Papadopoulos]

My wrong, the difference is between versions 3.0 dans 3.1, not between versions 2 and 3. So I'm rewriting comment #17:

Comment 8 describes how to enable/disable gloda indexing in Thunderbird 3.0.

Comment 14 explains I might have not enabled gloda indexing properly in
Thunderbird 3.1 so I might be comparing Thunderbird 3.0 and 3.1 with different
parameters.

How to enable/disable gloada indexing in Thunderbird 3.1?

Comment 20

[Ludovic Hirlimann [:Usul]]

There is no difference between 3.0 and 3.1 on disabling gloda.

Comment 21

[Dimitri Papadopoulos]

You're right and I don't understand my comment #13.

I've just launched Thunderbird 3.1.7 on Windows XP.
There is an item:
Tools | Options | Advanced | General | Advanced configuration | Enable global search and indexer
There is no item:
Options | General | Let Windows search messages
I don't understand where I got this last item from!

I'll check on the PC with the Gmail account whether gloda is enabled or not and let you know.

Comment 22

[Dimitri Papadopoulos]

I've just checked the Windows Vista machine with the Gmail account. Thunderbird 3.1.7 is using ~ 50 MB of memory and gloda is enabled - the check box is checked:
Tools | Options | Advanced | General | Advanced configuration | Enable global
search and indexer

This fixes the bug I had reported.  Thunderbird is still using much more memory than Windows Mail (~ 5 MB) but at least it is now usable in our case.

Comment 23

[Wayne Mery (:wsmwk)]

so perhaps this is gone in 1.9.2.  To take this a step further ...

Dimitri, can you also confirm this issue doesn't exist in trunk (3.3a) builds?
 ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-central/
first, backup your profile. and note, the gloda index will rebuild when you start it, and also when you go back to version 3.1.7.

Comment 24

[Wayne Mery (:wsmwk)]

Dimitri won't be able to test trunk for 3-4 weeks. And there is no testcase for asuth.  So let's close this WFM on the strength of comment 22. And you can reopen if you see the problem when using v3.1.10 or something newer.