1266378 - Need a tool to draw a fragmentation map of the content process.

Status: RESOLVED FIXED

(Core :: DMD, enhancement)

Description

[Jukka Jylänki]

Running asm.js pages with large heap sizes on 32-bit browsers has issues that even though there might be enough RAM available, it can happen that the process address space is too fragmented that the browser is not able to allocate such a large linear typed array for the heap of the asm.js module.

We have metrics from a partner developer who is shipping a major game title (in the range of millions of players) in asm.js, and they report in their crash statistics that this OOMing due to address space fragmentation is a serious concern.

There are some ideas to combat this, e.g. via process isolation for the asm.js module to guarantee a fresh address space, but that might not always be available.

Address space fragmentation is sometimes considered a "hard" problem, and there's a tendency to raise one's hands up at the face of it. However, I think it would be worthwhile to develop tooling to allow examining the structure of the native address space at the time of an OOM, which might help give ideas if there is something obvious that is trashing our address space that we could compact or free up.

The emunittest suite also suffers from OOMing due to address space fragmentation on 32-bit Firefox. It has several tests, and running any one of these tests after a fresh browser start will succeed, but running all of the tests in sequence will always cause an OOM in the middle. 

The DMD tool already keeps track of all allocated pointers and their sizes? That info could be used as a basis to paint a visual map of the address space that might help here? Has this kind of thing been done/attempted before?

Comment 1

[Nicholas Nethercote [inactive]]

DMD tracks heap blocks, but the heap is only part of a process's memory usage. There are also text segments, stacks, and any other mapped segments used for data purposes (e.g. the JS heap, JIT code).

On Linux you can look at /proc/self/maps or /proc/self/smaps to get a complete picture of the address space layout. Sometimes working out what each individual segment holds can be tricky, but it's a good place to start looking.

Comment 2

[(not currently active) Ted Mielczarek]

For Windows, VMMap is pretty useful for visualizing this info:
https://technet.microsoft.com/en-us/sysinternals/dd535533.aspx

Additionally, the minidumps we write from a crash on Windows include this information, if you look at the "Raw Dump" tab of a crash and search for "largest_free_vm_block" you can see that we summarize it. (Example crash that has this info: https://crash-stats.mozilla.com/report/index/c8b20ec9-a056-49ed-81f8-65ece2160415 ). n.b., we didn't have this information in e10s content process crashes until just recently, bug 1263774). The code that generates these stats is in Socorro's stackwalk.cc, it uses the Breakpad C++ APIs:
https://github.com/mozilla/socorro/blob/4796e8816448ef4d23fb53c0b3b86a97d330e167/minidump-stackwalk/stackwalker.cc#L709

For OS X, there's a commandline tool also helpfully called vmmap(1):
https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/vmmap.1.html#//apple_ref/doc/man/1/vmmap

Comment 3

[Eric Rahm [:erahm]]

I'm reasonably sure our Windows memory reports include details on fragmentation. I believe B2G (and maybe android) also include the data from smaps in the system memory report.

If you just want to look at the heap that DMD knows about there's mccr8's heap snapshot stuff that ties into DMD.

Comment 4

[Jukka Jylänki]

Thanks, I was able to use VMMap for this purpose, although it gave a bit opaque view to the data, not sure how well it reflected.

The need to have this kind of tool came from the perspective that we'd be tackling 32-bit address space fragmentation by explicitly looking which things are causing excessive fragmentation, and to see if we could identify places to use memory allocation schemes that wouldn't fragment as much, or if we had some opportunities to compact.

Since then, we have opted to process isolation as the solution to guarantee a fresh address space for these applications, which makes this kind of investigation not that important anymore, as it was verified that in a fresh address space we are able to allocate large contiguous blobs before Firefox fragments too much.