819771 - Add dark dark matter (memory used by anonymous mmap'ed pages that's not the heap or JS) detection to DMD

Status: NEW

(Core :: General, defect)

Description

[Justin Lebar (not reading bugmail)]

With newdmd, it should be trivial to intercept all calls to mmap and munmap.

Thus it shouldn't be too hard to extend DMD to track virtual allocations.

I would expect that writing "memory reporters" for virtual allocations would be annoying; we might not want JS and the heap to keep track of and report every block they own.

But I don't think that's entirely necessary.  Although we call malloc from many different places (and we're interested in the full stack which caused the call to malloc), there should be relatively few places in JS and our allocator where we call mmap, and we're not interested in much of the stack in these cases.

Therefore if we simply were to report all callsites which invoke mmap, that might not have so much noise as to require filtering, in the way that we have to filter DMD's output to show only unreported and double-reported blocks.

Comment 1

[Nicholas Nethercote [inactive]]

> With newdmd, it should be trivial to intercept all calls to mmap and munmap.

Really?  How?

Comment 2

[Justin Lebar (not reading bugmail)]

> Really?  How?

The same way libmozglue intercepts calls to malloc.  On Linux, this is just by declaring the symbol and exporting it.  Since we don't have smaps on other systems, getting this to work on Linux would be just fine.

(Note that we already have code on Windows which is intercepting calls to VirtualAlloc and VirtualFree, the equivalents of mmap and munmap; this code lives in AvailableMemoryTracker.cpp.  And Mac OS may function the same as Linux for our purposes here; it's more complex for the purposes of malloc because they have these malloc zones and so forth.)

Comment 3

[Mike Hommey [:glandium]]

(In reply to Justin Lebar [:jlebar] from comment #2)
> And Mac OS may function the same as Linux for our purposes here;

Probably not, unless you link with flat namespaces.

Comment 4

[Justin Lebar (not reading bugmail)]

(In reply to Mike Hommey [:glandium] from comment #3)
> (In reply to Justin Lebar [:jlebar] from comment #2)
> > And Mac OS may function the same as Linux for our purposes here;
> 
> Probably not, unless you link with flat namespaces.

Do we have to link libmozglue or libxul with flat namespaces?  Is there some reason we can't do that?

Comment 5

[Justin Lebar (not reading bugmail)]

> Do we have to link libmozglue or libxul with flat namespaces?

(Or I suppose for the purposes of this bug, s/libmozglue/libdmd.)

Comment 6

[Mike Hommey [:glandium]]

You'd have to build all libraries with flat namespace so that mmap/munmap can be interposed. Even then, you'd only catch mmap/munmap calls from gecko, not from system libraries. You'd have to force the dynamic linker to do flat namespaces. Also note mmap/munmap is probably not the whole story on OSX. There are various vm_* functions that can be used to map anonymous pages.

It might be worth investigating symbol interposing which would probably work better than flat namespaces. http://toves.freeshell.org/interpose/ I considered it for bug 804303 but it was too cumbersome for that use case. Here, it would probably work.

Comment 7

[Nicholas Nethercote [inactive]]

> The same way libmozglue intercepts calls to malloc.  On Linux, this is just
> by declaring the symbol and exporting it.

And hope that code isn't calling mmap/munmap directly via syscall()!

Comment 8

[Nicholas Nethercote [inactive]]

Another thing that makes mappings harder for DMD than heap blocks is that you can unmap part of a mapping.  To support this requires:

- the ability to split an existing mapping into multiple mappings
- the ability to lookup a mapping using any address within it (as opposed to its starting address)