Closed Bug 526568 Opened 15 years ago Closed 13 years ago

Consumes too much memory, nearly 2 GB with gloda indexing on, 800MB with gloda indexing off, many folders set to check for new maill

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: braden, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: perf, Whiteboard: [needs 3.1 retest of comment 28][quick filter?])

Attachments

(2 files)

msgdb log using 02-16-2010 nightly
58.99 KB, application/x-bzip2
Details
log from 2-16-2010 nightly:MSGDB:5,ImapAutoSync:5,timestamp
98.12 KB, application/x-bzip2
Details 
Memory consumption by Thunderbird continues to be *way* too high, even as of the 11/3 nightly. As I write this, Thunderbird is camping out on 1.76 GB. This is resident memory, not virtual. I've seen it get even closer to 2 GB at times.

I have a lot of IMAP folders; several of them large (i.e., tens of thousands of messages). Perhaps that is a contributing factor.
How big is the global-messages-db.sqlite file in your profile ?
Keywords: perf
(In reply to comment #1)
> How big is the global-messages-db.sqlite file in your profile ?

582 MB
Component: General → Backend
Product: Thunderbird → MailNews Core
QA Contact: general → backend
Version: 3.0 → Trunk
Xref bug  516395.

Can you set mail.check_all_imap_folders_for_new to false and see if your memory issue goes away ?
It does not. (Thunderbird is using 1.4 GB at the moment, running with that setting in user.js.)

Does this setting override individual folder settings? Because for most of my folders, I've checked the "Check this folder for new messages" box in the folder properties.

Like the reporter in bug 516395, I use server-side filtering of e-mail.
No, setting that pref to true is the same as checking the checkbox for all of your folders. If most of your folders are already checked, it's quite possible you're running into the same kind of problem.
Depends on: 516395
Braden(bug opener), do you use add-on such as Enigmail or Lightning?
Does your problem occur with -safe-mode of Tb?
Can you reproduce such huge real memory usage(and virtual too) problem with newly created profile? (Disable "Global Search and Indexer" for problem isolation).
(In reply to comment #6)
> Braden(bug opener), do you use add-on such as Enigmail or Lightning?

No.

> Does your problem occur with -safe-mode of Tb?

Yes.

> Can you reproduce such huge real memory usage(and virtual too) problem with
> newly created profile? (Disable "Global Search and Indexer" for problem
> isolation).

Yes.
(In reply to comment #0)
> As I write this, Thunderbird is camping out on 1.76 GB.
> This is resident memory, not virtual.
> I've seen it get even closer to 2 GB at times.

Questions to avoid going to wrong direction.
(Q1) How did you get the "resident memory size of Tb" of 1.76 GB or 2 GB?
(Q2) When "resident memory size"==1.76 GB(or 2 GB) was obtained, what was  "virtual memory size used by Tb"? How did you obtain the "virtual memory size used by Tb"?
(In reply to comment #8)
> (Q1) How did you get the "resident memory size of Tb" of 1.76 GB or 2 GB?

From the "Real Mem" column in the Activity Monitor.

> (Q2) When "resident memory size"==1.76 GB(or 2 GB) was obtained, what was 
> "virtual memory size used by Tb"?

I don't recall specifically; I think it was in the 3 GB ballpark.

> How did you obtain the "virtual memory size
> used by Tb"?

That would have been the "Virtual Mem" column in Activity Monitor.

Things may have improved somewhat since I disabled "Global Search and Indexer"; but they're still pretty bad. At the moment, Thunderbird is consuming a little over 800 MB of "Real Mem" and nearly 900 MB virtual.
> Mac OS X Activity Monitor
> http://support.apple.com/kb/HT1342
> VM size
> "VM size" is the amount hard disk space being used as virtual memory
> (this number also includes the amount of installed RAM).
> Virtual memory allows Mac OS X to "virtually" use more memory than the amount
> of RAM you have by using hard disk space to supplement RAM. However, hard
> disks are much slower than RAM, so Mac OS X automatically distributes
> information between disk space and RAM for efficient performance.

Mac OS X may allocate real-memory to allocated-but-recently-not-touched virtual-memory. It may produce larger real memory size report than working-set size Tb really requires. Check both virtual memory size and real memory size always.
Sorry but I don't know total virtual memory size allocated to an application is (Real Mem + Virtual Mem) or (Virtual Mem only) on Mac OS X.


(In reply to comment #1)
> > How big is the global-messages-db.sqlite file in your profile ?
> 582 MB
(In reply to comment #9)
> Things may have improved somewhat since I disabled "Global Search and Indexer";
> but they're still pretty bad. At the moment, Thunderbird is consuming a little
> over 800 MB of "Real Mem" and nearly 900 MB virtual.

(1.76 GB - 800 MB) == 900 MB. Delta seems memory for sqlite DB mainly.

Check file size of .msf files.  
Check Real Mem/Virtual Mem when several large IMAP folders are unsubscribed.
If Inbox is very large, and if possible, move many mains in Inbox to Inbox-2 folder, unsubscribe Inbox-2, compact Inbox, then check memory usage, please.
Check Real Mem/Virtual Mem for next cases, please.
 Case-1 : several large folders are unsubscribed
 Case-2 : several large folders are subscribed but offline use=off(no auto-sync),
          open the several large folders
 Case-3 : several large folders are subscribed and offline use=on(auto-sync on),
          open the several large folders
FYI.
Bug 516395 Comment #16 reports similar VM size and Real memory size on Linux, after disabling Gloda(Global Search and Indexer).
On Linux, it looks that VIRT==Total virtual memory size allocated to Tb && RES==real memory size used for the "Total virtual memory size".
Severity: normal → major
(In reply to comment #10)
> (In reply to comment #1)
> > > How big is the global-messages-db.sqlite file in your profile ?
> > 582 MB
> (In reply to comment #9)
> > Things may have improved somewhat since I disabled "Global Search and Indexer";
> > but they're still pretty bad. At the moment, Thunderbird is consuming a little
> > over 800 MB of "Real Mem" and nearly 900 MB virtual.
> 
> (1.76 GB - 800 MB) == 900 MB. Delta seems memory for sqlite DB mainly.

Well, after a couple more days of use, Thunderbird now takes up 1.26 GB real, 2.16 GB virtual.

> Check file size of .msf files.

I don't think they're uncharacteristically large. For instance, the INBOX folder itself has over 27k messages, and its .msf file is 14 MB.

> Check Real Mem/Virtual Mem when several large IMAP folders are unsubscribed.

If anything, that seems to have increased memory usage: now at 1.36 GB real, 2.79 GB virtual.

> If Inbox is very large, and if possible, move many mains in Inbox to Inbox-2
> folder, unsubscribe Inbox-2, compact Inbox, then check memory usage, please.

Sorry, moving large numbers of messages is more pain that I'm willing to incur for this.

I have Thunderbird set to "always start up online". I do not use offline mode.
Braden, can you try nightly 3.0.2 build to see if bug 540214 resolves your issue. 
  ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-1.9.1/

if it does not, then please work through the steps of https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems and report results. Thanks.
No longer depends on: 516395
After running the nightly for 15 minutes or so, it's already up to 730 MB real/782 MB virtual. That still seems pretty far out of bounds.

As I understand that wiki page, the next step here is to provide a db log; however, it sounds like I can't do that with 3.0.2pre.
recent 3.02pre builds do have db logging.
Apparently, I can. (Thanks, Wayne.)

msgdb log is here:

  http://endoframe.com/msgdb.log.bz2

This is from running the 2-16-2010 nightly for a few minutes, at which point memory usage seemed to settle consistently with what I described in comment #15.
that looks like an imap protocol log, not a MSGDB log, or an IMAP+MSGDB log.
Indeed, I goofed.

And, of course, the correct log is small enough to attach.
OK, thx, that's an interesting log. Something is opening a lot of imap folder db's, and instantiating some or all of the headers in the db. It would be interesting to know what's opening the db's and leaving them open. Gloda indexing is turned off during that log's run? I suspect autosync is opening the db's, assuming it's not you going around and clicking on each folder :-) Can you add ImapAutoSync to the log modules, so that you're logging both autosync and db opening/closing?
(In reply to comment #20)
> OK, thx, that's an interesting log. Something is opening a lot of imap folder
> db's, and instantiating some or all of the headers in the db. It would be
> interesting to know what's opening the db's and leaving them open. Gloda
> indexing is turned off during that log's run?

Yes.

> I suspect autosync is opening the
> db's, assuming it's not you going around and clicking on each folder :-)

A reasonable suspicion. mail.check_all_imap_folders_for_new is false; however, many folders are set individually to be checked for new messages.

> Can
> you add ImapAutoSync to the log modules, so that you're logging both autosync
> and db opening/closing?

Okay...
Braden, is there a high degree of correspondence between the folders you have set to be checked for new messages and the db's left open at the end of the log?
oh, and do you have the hidden pref mail.imap.use_status_for_biff set to false?
mail.imap.use_status_for_biff remains at its default, "true".

The ones at the very end? Yes, I believe so.

It's a bit confusing because now neither Thunderbird 3.0.1 nor 3.0.2pre appear to be retaining the per-folder "check for new messages" setting. That is, I can check the box for a folder and when I restart Thunderbird it will be unchecked again. It's not clear to me whether the actual setting's being mutated or if this is just a UI bug.

So, basically, I'm quite certain that I had quite a few folders that were set to be checked for new messages; however, now that doesn't *appear* to be set for any of my folders.
Braden, 
* how many folders have "Check this folder for new messages"?
* how does memory usage change if you turn disable "keep messages...on this computer" for the accounts that contain these folders?
* do you detect any slowness associated with high memory usage?
Summary: Consumes too much memory (nearly 2 GB) → Consumes too much memory, nearly 2 GB with gloda indexing on, 800MB with gloda indexing off
Braden wrote me "progress on this is rather hampered by bug 547032"
Depends on: 547032
One thing worth noting is that we discovered with the quick filter bar implementation that the quick search would actually end up holding onto nsIMsgDBHdr's.  So performing a quick search in a folder/virtual folder with a lot of messages could result in non-trivial memory bloat.
Braden, can you confirm that this issue is not related to quick search?
Braden?  much has changed since version 3.0. can you create a new log with version 3.1.6 or higher please?  And ...
>  can you confirm that this issue is not related to quick search?


FWIW, attachment 427178 [details] (3.0 2010-02-16 nightly) shows as many as 78 dbs open


(In reply to comment #21)
> (In reply to comment #20)
> > OK, thx, that's an interesting log. Something is opening a lot of imap folder
> > db's, and instantiating some or all of the headers in the db. It would be
> > interesting to know what's opening the db's and leaving them open. Gloda
> > indexing is turned off during that log's run?
> 
> Yes.
> 
> > I suspect autosync is opening the
> > db's, assuming it's not you going around and clicking on each folder :-)
> 
> A reasonable suspicion. mail.check_all_imap_folders_for_new is false; however,
> many folders are set individually to be checked for new messages.

in a quick look, I didn't find any related open bugs, and there aren't many major memory suckage bugs still open [1]. Though perhaps I missed something. 

[1] https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=substring&short_desc=memory&field0-0-0=short_desc&bug_severity=blocker&bug_severity=critical&bug_severity=major&type0-0-1=substring&field0-0-1=keywords&type1-0-1=allwordssubstr&resolution=---&classification=Client%20Software&classification=Components&query_format=advanced&short_desc_type=allwordssubstr&type0-0-0=anywordssubstr&field1-0-0=short_desc&product=MailNews%20Core&product=Thunderbird&field1-0-1=short_desc
Whiteboard: [needs 3.1 retest from braden]
From my point of view I did not experienced it in 3.1 ...
(In reply to comment #28)
> One thing worth noting is that we discovered with the quick filter bar
> implementation that the quick search would actually end up holding onto
> nsIMsgDBHdr's.  So performing a quick search in a folder/virtual folder with a
> lot of messages could result in non-trivial memory bloat.
Whiteboard: [needs 3.1 retest from braden] → [closeme 2011-03-01][needs 3.1 retest of comment 28][quick filter?]
no one else can reproduce, and needs more info.
please reopen when more info is available.
thanks
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → INCOMPLETE
Whiteboard: [closeme 2011-03-01][needs 3.1 retest of comment 28][quick filter?] → [needs 3.1 retest of comment 28][quick filter?]
(In reply to Wayne Mery (:wsmwk) from comment #30)
> Braden?  much has changed since version 3.0. can you create a new log with
> version 3.1.6 or higher please?  And ...
> >  can you confirm that this issue is not related to quick search?
> 
> 
> FWIW, attachment 427178 [details] (3.0 2010-02-16 nightly) shows as many as
> 78 dbs open

Or rather, 97 open dbs

bug 723248 limits open dbs to 30, which should help.  Check bug 723248 comment 29
Depends on: 723248
See Also: → 671613
Summary: Consumes too much memory, nearly 2 GB with gloda indexing on, 800MB with gloda indexing off → Consumes too much memory, nearly 2 GB with gloda indexing on, 800MB with gloda indexing off, many folders set to check for new maill
WFM based on Braden's bug 671613 comment 19.  (cause/solution unknown)
Resolution: INCOMPLETE → WORKSFORME
Blocks: 1330872
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: