516395 - Extremely high memory usage under normal operations and gloda indexing is OFF - folder db not closed on exit

Status: RESOLVED DUPLICATE of bug 540214

(MailNews Core :: Backend, defect)

Description

[Karanbir Singh]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.0.11) Gecko/2009061613 CentOS/3.0.11-2.el5.centos Firefox/3.0.11
Build Identifier: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.4pre) Gecko/20090910 Shredder/3.0b4pre

Thunderbird is using in excess of 1.5GB of ram, even when there is no user interaction going on. 


Reproducible: Always

Steps to Reproduce:
1.Start thunderbird

Actual Results:  
1.Within 2 - 3 minutes used memory reaches 800MB
2.Within 12 - 14 minutes, used memory reaches 1.3GB
3.Within 30 minutes, used memory reaches ~ 1.8GB

eg:
---------
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                
 4615 kbsingh   15   0 1726m 1.1g  25m R   45 13.7  90:44.90 thunderbird-bin     
---------

( there is no need to actually do anything, open emails or even move emails around, just start Thunderbird, let it login to the imap accounts and just watch memory used )

When doing 'stressing' work, like moving 2,000+ emails, memory used jumps to over 2.4GB, however does come back to the usual 1.6 - 1.8GB within a few minutes.

Expected Results:  
A desktop email client should be expected to never use more than 300MB of memory.

Turning on/off adaptive Junk filtering has no impact.

There are 4 email accounts setup, all over imap. 
Account-1: 4.5GB of email, with ~ 700 unread in INBOX ( of a total of 5,000 emails in INBOX ). Account has ~ 36,000 emails spread over 75 folders. 
Account-2: 7.5GB of email, with no unread in UNBOX ( of a total of 17,000 in INBOX ). Account has ~ 317,000 emails spread over 160 folders.
Acount-3 and Account-4 have nothing much ( ~ 3000  - 3500 emails spread across 5 - 8 folders, and get less than 3 emails/day )

cd ~/.thunderbird/ ; du -sh => 19GB

replicated on :

1) x86_64 host, running x86_64 build ( for testing purpose )
2) x86_64 host, running a i686 build ( primary desktop )
3) i686 host running a i686 build ( laptop )

I'll be happy to provide any other info that might be needed.

Comment 1

[Ludovic Hirlimann [:Usul]]

Did it start with build Gecko/20090910 ?

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

Thank you for including the full ps output, it's good to be able to see the resident size and not just the virtual size.  For the sake of discussion, I think we should stick to using resident/working-set numbers.

The memory usage you cite is pretty high.  Assuming your INBOX with 17,000 messages is the largest single folder you have (please confirm), this would suggest that for some reason we are opening other folders and holding them open.

Do you have any custom virtual folders/saved search folders that span multiple folders?

Are you using *any* extensions?  Extensions may be peeking into your other folders and not cleaning up after themselves.

Comment 3

[Karanbir Singh]

(In reply to comment #1)
> Did it start with build Gecko/20090910 ?

I've had this issue for a while ( 12 months + ) but its not been that big a deal since my main desktop has 8GB of ram, the memory usage wase'nt 'causing problems'. Its only in the last few weeks that I've wanted to get email going on the laptop ( which only had a total of 1GB ), that this started getting in the way. I've since upgraded ram on the laptop to 2G ( the max it can take ), and as you can see, that's clearly not enough. 

I dont think its unreasonable to expect thunderbird + firefox + eclipse to work at the same time, on a Turion 2Ghz laptop with 2 GB of ram!

Comment 4

[Karanbir Singh]

(In reply to comment #2)
> The memory usage you cite is pretty high.  Assuming your INBOX with 17,000
> messages is the largest single folder you have (please confirm), this would
> suggest that for some reason we are opening other folders and holding them
> open.

so here is my attempt to try and quantify 'large':

$ cd ~/.thunderbird/ond77673.default/ImapMail
$ find . -type f -exec ls -l {} \; |  cut -f5,9 -d' ' | sort -nr |  head -n 10
2985404757 ./imap.gmail.com/[Google
2581243055 ./imap.gmail.com/Cvs
1475612193 ./monk/INBOX.sbd/CentOS.sbd/rpm-cvs
682968142 ./monk/INBOX.sbd/Tracker.sbd/logs
583704721 ./monk/INBOX.sbd/Fdr.sbd/devel
358271262 ./monk/INBOX.sbd/Xen.sbd/devel
312795869 ./monk/INBOX.sbd/Fdr.sbd/Test
290585605 ./monk/INBOX.sbd/Trash
258888768 ./monk/INBOX.sbd/ppc.sbd/linux
241474847 ./monk/INBOX.sbd/Sent

The other dimension to large being number of emails, so:
$ for f in `find . -type f | grep -v .msf | grep -v .dat `; do echo -n $( grep 'From - ' $f  | wc -l ); echo ' ' $f ; done | sort  -nr | head -n10
94518  ./monk/INBOX.sbd/Fdr.sbd/devel
48784  ./monk/INBOX.sbd/CentOS.sbd/c_old
47600  ./monk/INBOX.sbd/Tracker.sbd/logs
46198  ./monk/INBOX.sbd/Fdr.sbd/Test
41461  ./imap.gmail.com/Cvs
40520  ./monk/INBOX.sbd/CentOS.sbd/nag
34299  ./monk/INBOX.sbd/Xen.sbd/devel
33146  ./monk/INBOX.sbd/ppc.sbd/linux
30826  ./monk/INBOX.sbd/Xen.sbd/Users
27908  ./monk/INBOX.sbd/glug

( my initial estimate of 317k emails in this mailbox is clearly out by a large number - the actual is closer to 680k )

> Do you have any custom virtual folders/saved search folders that span multiple
> folders?

Not by design. I've not setup any saved search's or virtual folders.

> Are you using *any* extensions?  Extensions may be peeking into your other
> folders and not cleaning up after themselves.

Only extension setup and working at this end ( i686 build on x86_64 host ) is sieve-0.1.6 : which should not be doing anything with the local mail store.

The only other extension I've ever really used, over the years, is sync-on-arrival, which is clearly no longer needed.

I wonder if this will help:
# /usr/sbin/lsof | grep thunder | grep ImapMail | wc -l
285

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

(In reply to comment #4)
> I wonder if this will help:
> # /usr/sbin/lsof | grep thunder | grep ImapMail | wc -l
> 285

While you have a truly impressive number of messages in general, this sounds like the problem.  We should not have all those folders open.

A brief survey of the sieve code suggests it is unlikely to contribute to the problem, as you suggest.  I would expect autosync or gloda indexing is going off the rails somehow.

I'm not completely up on where the build identifiers come from... for your custom build, is your tree up-to-date-ish or mozilla-provided nightlies (that should be up-to-date)?  Sep 10 is fine, but something more than a few weeks old might have bugs.

Comment 6

[Karanbir Singh]

I've had global.indexer disabled from the configs - so it should not be running ( I can see the sqlite files are only stubs and not being updated ). If you could recommend changes to disable the autosync, I can try turning those off as well. At the moment I have :

mail.server.default.autosync_max_age_days | default | integer | -1
mail.server.default.autosync_offline_Stores | default | boolean | true

for search:

mailnews.database.global.indexer.enabled | user set | boolean | false

w.r.t builds, I am using the nightlies from mozilla. These might be relevant:
app.update.auto | default | boolean | true
app.update.channel | default | string | nightly
app.update.enabled | default | boolean | true
app.update.idletime | default | integer | 60
app.update.url | default | string |  https://aus2.mozillamessaging.com/update/3/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml
app.update.url.details | default | string | http://www.mozillamessaging.com/%LOCALE%/%APP%/releases/

Comment 7

[Karanbir Singh]

autoupdated to : Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.4pre) Gecko/20090913 Shredder/3.0b4pre

issue persists

Comment 8

[Wayne Mery (:wsmwk)]

Karanbir, do you see this problem when using ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-1.9.1/

Comment 9

[Karanbir Singh]

(In reply to comment #8)

Upgraded to : Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.5pre) Gecko/20091012 Shredder/3.0pre

After running ( and actively using email ) for about 15 minutes or so I have:
  
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 5148 kbsingh   15   0 1202m 824m  23m S   65 10.3   2:38.61 thunderbird-bin    


I can confirm that gloda indexing is still turned OFF. 

I have also now completely removed my ~/.thunderbird/ and resetup a new profile - hopefully that would clear out any stale profile specific things/settings hanging around. It will take a few hours for my emails to sync over from the imap server again, once its done I'll update this issue with the status.

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> When doing 'stressing' work, like moving 2,000+ emails,
> memory used jumps to over 2.4GB,
> however does come back to the usual 1.6 - 1.8GB within a few minutes.

Similar phenomenon to Bug 519226 Comment #2? (Large VM size)
Will "open new Tb window + close old Tb window" reduce virtual memory size used by Tb?
Or similr phenomenon to Bug 452221? (High CPU and UI lock. Large VM if many mails are deleted/moved)

Comment 11

[Dave Miller [:justdave]]

I was pointed at this bug from IRC, and someone suggested setting mail.check_all_imap_folders_for_new to false, and this indeed fixed the memory problem for me.  However, it also means I don't get notified when new mail shows up in folders other than the inbox (because my filters are all server-side).

Can the reporter of this bug try that out and see if that affects yours that way also?  If it does, then that makes this problem something to do with the new mail checks...

Comment 12

[Karanbir Singh]

Turning off mail.check_all_imap_folders_for_new does seem to make a difference. After running for a few hrs I now have :  

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND             
31741 kbsingh   15   0 1214m 161m  27m S  0.0  2.8  33:43.34 thunderbird-bin

Running: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.6pre) Gecko/20091108 Shredder/3.0pre

I've measured this over a course of a few days and the RES component rarely goes over 300m

Comment 13

[Dave Miller [:justdave]]

Yeah, in the long run it didn't help me either, memory was fine until I tried to actually look at a few folders, and the memory use went up as I opened them, and then stayed up.

Comment 14

[Wayne Mery (:wsmwk)]

bienvenu, looks like 2 good testcases.

[keeping bug 526568 unduped for now, preemptively setting in-testsuite?]

Comment 15

[Wayne Mery (:wsmwk)]

can you test and reproduce started in safe mode?
 https://support.mozilla.com/en-US/kb/Safe+Mode

Comment 16

[Karanbir Singh]

Running: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.7) Gecko/20091230 Shredder/3.0.1pre

in safe mode: 
   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 6907 kbsingh   15   0 1186m 747m  21m S    2  9.4   2:05.63 thunderbird-bin 

And this is only a few minutes since starting up ( about 15min wall clock )

Comment 17

[Wayne Mery (:wsmwk)]

Karanbir, do you also see bug 505577?

Comment 18

[Karanbir Singh]

(In reply to comment #17)
> Karanbir, do you also see bug 505577?

I'm not using the newsreader in thunderbird ( so no nntp setup at all ). And I've not really noticed any form of slowdown scrolling folders etc.

There are times when writing a new email thunderbird's response will slow right now - almost to the level where I can type much faster than its able to display. Given that I am now averaging > 1GiB of ram used, its hard to workout if the slowdown is related to a sudden increase in ram usage.

In specific terms though : no, I dont see a slowdown in 'scrolling through folders'. I do have more than 200 folders in all.

Comment 19

[Wayne Mery (:wsmwk)]

bienevenu, asuth ... what's the next step - do we need a debug build for Karanbir + justdave?

Karanbir. thanks for the feedback. to clarify ... the issue of bug 505577 isn't primarily or just news. and, it is more evident when tested immediately after startup, and reliably found only by examining the memory usage, not scroll behavior.

Comment 20

[Andrew Sutherland [:asuth] (he/him)]

Memory reporters as suggested on bug 480843 would probably be useful.  We don't really have a way to gather metrics other than trying to infer from the number of / which files are open.  (Note that memory reporters wouldn't tell us which files are open, but we could at least differentiate between nntp/local/imap and maybe inbox if we maintained enough distinct counters.)

Comment 21

[Wayne Mery (:wsmwk)]

please use instructions at https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems to get a msgdb log

Comment 22

[David :Bienvenu]

That logging is only available in 3.1b1 pre nightly builds.

Comment 23

[David :Bienvenu]

Have you tried today's nightly trunk build? It has a fix for bug 540214, which caused similarly high memory usage.

Comment 24

[Karanbir Singh]

I am just installing the nightly now, will have stats and a report by close of play today. thanks

Comment 25

[David :Bienvenu]

not blocking until we get more info steps to reproduce this (and confirmation that it's still happening).

Comment 26

[Karanbir Singh]

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                         
22091 kbsingh   21   0  695m 256m  28m R   95  3.2  17:25.56 thunderbird-bin                 

this is after just a short while of usage ( ~ 25 min ). But the situation does clearly look to have improved. The other problem now is the 90% CPU usage that it sits on constantly. I'll open a seperate bug report for that.