Closed Bug 696690 Opened 8 years ago Closed 8 years ago

Memory reporter for the startup cache

Categories

(Core :: XPCOM, defect)

defect
Not set

Tracking

()

RESOLVED FIXED
mozilla10

People

(Reporter: njn, Assigned: njn)

References

Details

(Whiteboard: [MemShrink])

Attachments

(1 file, 1 obsolete file)

The startup cache should have one or more memory reporters.  In particular, I saw this once I extended DMD to track mmap/munmap:

==1586== Unreported: 958,464 (cumulative: 535,731,680) bytes in 1 mapped block(s) in record 31 of 15105:
==1586==  Requested bytes unreported: 958,464 / 958,464
==1586==  Slop      bytes unreported: 0 / 0
==1586==    at 0x5228D6A: mmap (syscall-template.S:82)
==1586==    by 0x4089FC1: _MD_MemMap (unix.c:3612)
==1586==    by 0x4066BF3: PR_MemMap (prmmap.c:82)
==1586==    by 0x6855720: nsZipHandle::Init(nsILocalFile*, nsZipHandle**) (nsZipArchive.cpp:194)
==1586==    by 0x6855E83: nsZipArchive::OpenArchive(nsIFile*) (nsZipArchive.cpp:296) 
==1586==    by 0x6865D63: mozilla::scache::StartupCache::LoadArchive() (StartupCache.cpp:225)
==1586==    by 0x6865C36: mozilla::scache::StartupCache::Init() (StartupCache.cpp:201)
==1586==    by 0x686555F: mozilla::scache::StartupCache::InitSingleton() (StartupCache.cpp:111)

That's big enough to be worth tracking.  We've been using the term "dark matter" to refer to memory covered by about:memory's "heap-unclassified" entry, which is a "known unknown".  In contrast, this could be called "darker matter" -- it's not included in "heap-unclassified" and so is part of an "unknown unknown".

(BTW: as far as I can tell the above allocation lives for the life of the browser.  Is this intended?  The "startup cache" sounds like something that might be unloadable after a while, but I'm just guessing really.)

I also saw the following, which isn't terribly big:

==1586== Unreported: 126,976 (cumulative: 548,427,584) bytes in 2 heap block(s) in record 79 of 15105:
==1586==  Requested bytes unreported: 122,444 / 122,444
==1586==  Slop      bytes unreported: 4,532 / 4,532
==1586==    at 0x402A063: malloc (vg_replace_malloc.c:263) 
==1586==    by 0x403BFFA: moz_xmalloc (mozalloc.cpp:103)
==1586==    by 0x68660B8: mozilla::scache::StartupCache::PutBuffer(char const*, char const*, unsigned int) (mozalloc.h:242)
==1586==    by 0x74A4088: WriteCachedScript(mozilla::scache::StartupCache*, nsACString_internal&, JSContext*, JSScript*) (moz
JSLoaderUtils.cpp:189)
==1586==    by 0x749F1D5: mozJSComponentLoader::GlobalForLocation(nsILocalFile*, nsIURI*, JSObject**, char**, JS::Value*) (mo
zJSComponentLoader.cpp:1003) 
==1586==    by 0x749D77F: mozJSComponentLoader::LoadModuleImpl(nsILocalFile*, nsAString_internal&, nsIURI*) (mozJSComponentLo
ader.cpp:607)
==1586==    by 0x749D3A4: mozJSComponentLoader::LoadModule(nsILocalFile*) (mozJSComponentLoader.cpp:547)
==1586==    by 0x7A42B50: nsComponentManagerImpl::KnownModule::Load() (nsComponentManager.cpp:951)

I've seen other startup cache entries in DMD's output on occasion, but they tend to be short-lived.
Attached patch patch (obsolete) — Splinter Review
This patch adds a "startup-cache" reporter which measures the size of the file mapped into memory.

We could also measure StartupCache::mTable and the strings that it points to, but I haven't done that.  Maybe in a follow-up bug.
Assignee: nobody → nnethercote
Attachment #568997 - Flags: review?(tglek)
Nicholas,
This mmap() memory usage is similar to memory overhead of libraries so shouldn't this be resident/startup-cache rather than explicit(and we could also read proc/smaps to have a more exact figure)?

The second one is indeed explicit and worth tracking.
Comment on attachment 568997 [details] [diff] [review]
patch

startup cache mapping is mainly accessed on startup, OS should page most of it out once browser is started up. Reporting that as an explicit allocation is misleading
Attachment #568997 - Flags: review?(tglek) → review-
> startup cache mapping is mainly accessed on startup, OS should page most of it out once browser is 
> started up.

It would be great if we could test whether this is true.  It looks like the file being mapped should be named "startupCache", but I don't see that in about:memory's smaps breakdown.  What's the file called on Linux?

> This mmap() memory usage is similar to memory overhead of libraries so shouldn't this be 
> resident/startup-cache rather than explicit

We don't have a resident/ tree.
(In reply to Justin Lebar [:jlebar] from comment #4)
> > startup cache mapping is mainly accessed on startup, OS should page most of it out once browser is 
> > started up.
> 
> It would be great if we could test whether this is true.  It looks like the
> file being mapped should be named "startupCache", but I don't see that in
> about:memory's smaps breakdown.  What's the file called on Linux?

startupCache.4.little on 32bit, .8.little on 64. Note that startupCache is folded into the omnijar on PGO builds. 

I'm guessing about:memory doesn't track our mmap()ed zip files either.

> 
> > This mmap() memory usage is similar to memory overhead of libraries so shouldn't this be 
> > resident/startup-cache rather than explicit
> 
> We don't have a resident/ tree.

http://blog.mozilla.com/nnethercote/2011/09/07/memshrink-progress-week-12/ shows resident memory.
So I guess I should have said "we don't build a resident/ tree using memory reporters."  :)
(In reply to Justin Lebar [:jlebar] from comment #7)
> So I guess I should have said "we don't build a resident/ tree using memory
> reporters."  :)

The right fix is to show startup cache(and other mmaped files in the profile dir) under resident/
resident/ only exists on Linux, but startupcache exists elsewhere (?).  What do you propose we do outside Linux?

I just did a non-PGO release build, and I don't see omnijar or startupcache in my vsize or RSS breakdown.  Maybe we're interpreting these as anonymous mappings.  That would be lame.
(In reply to Justin Lebar [:jlebar] from comment #9)
> resident/ only exists on Linux, but startupcache exists elsewhere (?).  What
> do you propose we do outside Linux?
> 
> I just did a non-PGO release build, and I don't see omnijar or startupcache
> in my vsize or RSS breakdown.  Maybe we're interpreting these as anonymous
> mappings.  That would be lame.

no omnijar in dist/bin, you'll see omnijar if you make package and run from dist/firefox. Re outside of linux: same thing(ie nothing) as we do for library files(since their overhead is greater anyway).
(In reply to Taras Glek (:taras) from comment #3)
> 
> startup cache mapping is mainly accessed on startup, OS should page most of
> it out once browser is started up. Reporting that as an explicit allocation
> is misleading

No.  I defined the term "explicit";  it's for all allocations directly done by Firefox, including all calls to malloc and mmap.

I do not want to ignore some allocations.  It's better to include the startup mapping.  Then it's possible to say "oh, that doesn't matter much because it's only accessed at startup".
Attached patch patch, v2Splinter Review
I talked with Taras on IRC, he's now happier about this being included under "explicit".  I updated the reporter's description, too.
Attachment #568997 - Attachment is obsolete: true
Attachment #569239 - Flags: review?(tglek)
(In reply to Taras Glek (:taras) from comment #5)
> 
> I'm guessing about:memory doesn't track our mmap()ed zip files either.

Which files are they?  I didn't see any other significant mmap'd files in my DMD output, but there could be cases that I didn't see in that run.
Comment on attachment 569239 [details] [diff] [review]
patch, v2

> 
>+static PRInt64
>+GetStartupCacheSize()
>+{
>+    return mozilla::scache::StartupCache::GetSingleton()->SizeOfMapping();
>+}

Note this can fail if there is no profile(ProfLDS) dir.
Attachment #569239 - Flags: review?(tglek) → review+
(In reply to Nicholas Nethercote [:njn] from comment #13)
> (In reply to Taras Glek (:taras) from comment #5)
> > 
> > I'm guessing about:memory doesn't track our mmap()ed zip files either.
> 
> Which files are they?  I didn't see any other significant mmap'd files in my
> DMD output, but there could be cases that I didn't see in that run.

extensions are kept in zips + omni.jar in dist/firefox.
Man, that's the third time I've seen people have that exact same bustage on Maemo. (semicolon after a macro)
https://hg.mozilla.org/mozilla-central/rev/a394d649cf90
https://hg.mozilla.org/mozilla-central/rev/c20d0c530997
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla10
Blocks: 697335
(In reply to Taras Glek (:taras) from comment #15)
> (In reply to Nicholas Nethercote [:njn] from comment #13)
> > (In reply to Taras Glek (:taras) from comment #5)
> > > 
> > > I'm guessing about:memory doesn't track our mmap()ed zip files either.
> > 
> > Which files are they?  I didn't see any other significant mmap'd files in my
> > DMD output, but there could be cases that I didn't see in that run.
> 
> extensions are kept in zips + omni.jar in dist/firefox.

Is there a bug for this?
You need to log in before you can comment on or make changes to this bug.