The default bug view has changed. See this FAQ.

Memory reporter for the startup cache

RESOLVED FIXED in mozilla10

Status

()

Core
XPCOM
RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: njn, Assigned: njn)

Tracking

unspecified
mozilla10
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [MemShrink])

Attachments

(1 attachment, 1 obsolete attachment)

7.93 KB, patch
(dormant account)
: review+
Details | Diff | Splinter Review
(Assignee)

Description

6 years ago
The startup cache should have one or more memory reporters.  In particular, I saw this once I extended DMD to track mmap/munmap:

==1586== Unreported: 958,464 (cumulative: 535,731,680) bytes in 1 mapped block(s) in record 31 of 15105:
==1586==  Requested bytes unreported: 958,464 / 958,464
==1586==  Slop      bytes unreported: 0 / 0
==1586==    at 0x5228D6A: mmap (syscall-template.S:82)
==1586==    by 0x4089FC1: _MD_MemMap (unix.c:3612)
==1586==    by 0x4066BF3: PR_MemMap (prmmap.c:82)
==1586==    by 0x6855720: nsZipHandle::Init(nsILocalFile*, nsZipHandle**) (nsZipArchive.cpp:194)
==1586==    by 0x6855E83: nsZipArchive::OpenArchive(nsIFile*) (nsZipArchive.cpp:296) 
==1586==    by 0x6865D63: mozilla::scache::StartupCache::LoadArchive() (StartupCache.cpp:225)
==1586==    by 0x6865C36: mozilla::scache::StartupCache::Init() (StartupCache.cpp:201)
==1586==    by 0x686555F: mozilla::scache::StartupCache::InitSingleton() (StartupCache.cpp:111)

That's big enough to be worth tracking.  We've been using the term "dark matter" to refer to memory covered by about:memory's "heap-unclassified" entry, which is a "known unknown".  In contrast, this could be called "darker matter" -- it's not included in "heap-unclassified" and so is part of an "unknown unknown".

(BTW: as far as I can tell the above allocation lives for the life of the browser.  Is this intended?  The "startup cache" sounds like something that might be unloadable after a while, but I'm just guessing really.)

I also saw the following, which isn't terribly big:

==1586== Unreported: 126,976 (cumulative: 548,427,584) bytes in 2 heap block(s) in record 79 of 15105:
==1586==  Requested bytes unreported: 122,444 / 122,444
==1586==  Slop      bytes unreported: 4,532 / 4,532
==1586==    at 0x402A063: malloc (vg_replace_malloc.c:263) 
==1586==    by 0x403BFFA: moz_xmalloc (mozalloc.cpp:103)
==1586==    by 0x68660B8: mozilla::scache::StartupCache::PutBuffer(char const*, char const*, unsigned int) (mozalloc.h:242)
==1586==    by 0x74A4088: WriteCachedScript(mozilla::scache::StartupCache*, nsACString_internal&, JSContext*, JSScript*) (moz
JSLoaderUtils.cpp:189)
==1586==    by 0x749F1D5: mozJSComponentLoader::GlobalForLocation(nsILocalFile*, nsIURI*, JSObject**, char**, JS::Value*) (mo
zJSComponentLoader.cpp:1003) 
==1586==    by 0x749D77F: mozJSComponentLoader::LoadModuleImpl(nsILocalFile*, nsAString_internal&, nsIURI*) (mozJSComponentLo
ader.cpp:607)
==1586==    by 0x749D3A4: mozJSComponentLoader::LoadModule(nsILocalFile*) (mozJSComponentLoader.cpp:547)
==1586==    by 0x7A42B50: nsComponentManagerImpl::KnownModule::Load() (nsComponentManager.cpp:951)

I've seen other startup cache entries in DMD's output on occasion, but they tend to be short-lived.
(Assignee)

Comment 1

6 years ago
Created attachment 568997 [details] [diff] [review]
patch

This patch adds a "startup-cache" reporter which measures the size of the file mapped into memory.

We could also measure StartupCache::mTable and the strings that it points to, but I haven't done that.  Maybe in a follow-up bug.
Assignee: nobody → nnethercote
Attachment #568997 - Flags: review?(tglek)

Comment 2

6 years ago
Nicholas,
This mmap() memory usage is similar to memory overhead of libraries so shouldn't this be resident/startup-cache rather than explicit(and we could also read proc/smaps to have a more exact figure)?

The second one is indeed explicit and worth tracking.

Comment 3

6 years ago
Comment on attachment 568997 [details] [diff] [review]
patch

startup cache mapping is mainly accessed on startup, OS should page most of it out once browser is started up. Reporting that as an explicit allocation is misleading
Attachment #568997 - Flags: review?(tglek) → review-
> startup cache mapping is mainly accessed on startup, OS should page most of it out once browser is 
> started up.

It would be great if we could test whether this is true.  It looks like the file being mapped should be named "startupCache", but I don't see that in about:memory's smaps breakdown.  What's the file called on Linux?

> This mmap() memory usage is similar to memory overhead of libraries so shouldn't this be 
> resident/startup-cache rather than explicit

We don't have a resident/ tree.

Comment 5

6 years ago
(In reply to Justin Lebar [:jlebar] from comment #4)
> > startup cache mapping is mainly accessed on startup, OS should page most of it out once browser is 
> > started up.
> 
> It would be great if we could test whether this is true.  It looks like the
> file being mapped should be named "startupCache", but I don't see that in
> about:memory's smaps breakdown.  What's the file called on Linux?

startupCache.4.little on 32bit, .8.little on 64. Note that startupCache is folded into the omnijar on PGO builds. 

I'm guessing about:memory doesn't track our mmap()ed zip files either.

> 
> > This mmap() memory usage is similar to memory overhead of libraries so shouldn't this be 
> > resident/startup-cache rather than explicit
> 
> We don't have a resident/ tree.

http://blog.mozilla.com/nnethercote/2011/09/07/memshrink-progress-week-12/ shows resident memory.
> http://blog.mozilla.com/nnethercote/2011/09/07/memshrink-progress-week-12/ shows resident memory.

That comes straight from smaps.
So I guess I should have said "we don't build a resident/ tree using memory reporters."  :)

Comment 8

6 years ago
(In reply to Justin Lebar [:jlebar] from comment #7)
> So I guess I should have said "we don't build a resident/ tree using memory
> reporters."  :)

The right fix is to show startup cache(and other mmaped files in the profile dir) under resident/
resident/ only exists on Linux, but startupcache exists elsewhere (?).  What do you propose we do outside Linux?

I just did a non-PGO release build, and I don't see omnijar or startupcache in my vsize or RSS breakdown.  Maybe we're interpreting these as anonymous mappings.  That would be lame.

Comment 10

6 years ago
(In reply to Justin Lebar [:jlebar] from comment #9)
> resident/ only exists on Linux, but startupcache exists elsewhere (?).  What
> do you propose we do outside Linux?
> 
> I just did a non-PGO release build, and I don't see omnijar or startupcache
> in my vsize or RSS breakdown.  Maybe we're interpreting these as anonymous
> mappings.  That would be lame.

no omnijar in dist/bin, you'll see omnijar if you make package and run from dist/firefox. Re outside of linux: same thing(ie nothing) as we do for library files(since their overhead is greater anyway).
(Assignee)

Comment 11

6 years ago
(In reply to Taras Glek (:taras) from comment #3)
> 
> startup cache mapping is mainly accessed on startup, OS should page most of
> it out once browser is started up. Reporting that as an explicit allocation
> is misleading

No.  I defined the term "explicit";  it's for all allocations directly done by Firefox, including all calls to malloc and mmap.

I do not want to ignore some allocations.  It's better to include the startup mapping.  Then it's possible to say "oh, that doesn't matter much because it's only accessed at startup".
(Assignee)

Comment 12

6 years ago
Created attachment 569239 [details] [diff] [review]
patch, v2

I talked with Taras on IRC, he's now happier about this being included under "explicit".  I updated the reporter's description, too.
Attachment #568997 - Attachment is obsolete: true
Attachment #569239 - Flags: review?(tglek)
(Assignee)

Comment 13

6 years ago
(In reply to Taras Glek (:taras) from comment #5)
> 
> I'm guessing about:memory doesn't track our mmap()ed zip files either.

Which files are they?  I didn't see any other significant mmap'd files in my DMD output, but there could be cases that I didn't see in that run.

Comment 14

6 years ago
Comment on attachment 569239 [details] [diff] [review]
patch, v2

> 
>+static PRInt64
>+GetStartupCacheSize()
>+{
>+    return mozilla::scache::StartupCache::GetSingleton()->SizeOfMapping();
>+}

Note this can fail if there is no profile(ProfLDS) dir.
Attachment #569239 - Flags: review?(tglek) → review+

Comment 15

6 years ago
(In reply to Nicholas Nethercote [:njn] from comment #13)
> (In reply to Taras Glek (:taras) from comment #5)
> > 
> > I'm guessing about:memory doesn't track our mmap()ed zip files either.
> 
> Which files are they?  I didn't see any other significant mmap'd files in my
> DMD output, but there could be cases that I didn't see in that run.

extensions are kept in zips + omni.jar in dist/firefox.
(Assignee)

Comment 16

6 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/a394d649cf90
(Assignee)

Comment 17

6 years ago
Bustage fix for Maemo:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c20d0c530997
Man, that's the third time I've seen people have that exact same bustage on Maemo. (semicolon after a macro)
https://hg.mozilla.org/mozilla-central/rev/a394d649cf90
https://hg.mozilla.org/mozilla-central/rev/c20d0c530997
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla10
(Assignee)

Updated

6 years ago
Blocks: 697335

Comment 20

6 years ago
(In reply to Taras Glek (:taras) from comment #15)
> (In reply to Nicholas Nethercote [:njn] from comment #13)
> > (In reply to Taras Glek (:taras) from comment #5)
> > > 
> > > I'm guessing about:memory doesn't track our mmap()ed zip files either.
> > 
> > Which files are they?  I didn't see any other significant mmap'd files in my
> > DMD output, but there could be cases that I didn't see in that run.
> 
> extensions are kept in zips + omni.jar in dist/firefox.

Is there a bug for this?
You need to log in before you can comment on or make changes to this bug.