747605 - Add a heap profiler feature to Gecko Profiler

Status: NEW

(Core :: Gecko Profiler, enhancement, P5)

Description

[Benoit Girard (:BenWa)]

We already have a significant infrastructure for profiling code performance cross-platform (profiling, symbolication, add-on control, front end presentation, js integration soon) that's been designed to be modular and accept optional feature. Currently all feature are performance related and selected in the add-on but it's doesn't have to be that way.

I'd like to suggest that we add heap profiling features. We should add a few hooks to jemalloc on MOZ_PROFILING builds that do a cheap return when the heap profiling is disabled. When it's enabled there's a few things we feature we can provide:

1) Electric fence styles feature to detect out of bounds read/write, usage after free: http://en.wikipedia.org/wiki/Electric_Fence
2) Liveness: We randomly choose to track a block being allocated in jemalloc. Tracked block get no read/write permissions. When they segfault we set them as read/write and record the last use time. Every minute (or the desired resolution) we reset to no read/write on the page giving us a rough estimate within a minute when the page was last used. We can unwind the allocation/last usage point/dealloc. With this info we can find memory we're leaking, or holding on for far too long in caches.

The goal here is not to replace valgrind but to provide some debugging feature that are cross-platform and work with nearly no performance overhead, say < 5%. We can reduce the overheads by profiling subsets of allocations based on a probabilities. Power users with profiling extensions, such as myself, will always be collecting performance and heap data (I already run profiling 24/7 to catch small janks). The benefit is that the data is collected in a realistic browsing usage. Even if it only catches a subset of errors by having people run them all the time we can collect a lot of useful data. Again we're not looking to replace valgrind but provide ways for people to collect data in their regular browsing without any hassle.

Comment 1

[(no longer active)]

Seems like Address Sanitizer already takes care of the first feature above (on Linux and OS X, of course) and more.  Do you think it's useful to have our own version of this functionality as well?

Comment 2

[Justin Lebar (not reading bugmail)]

> 1) Electric fence styles feature to detect out of bounds read/write, usage after free: 
> http://en.wikipedia.org/wiki/Electric_Fence

I wouldn't necessarily call this "heap profiling".

In any case, you should look at jemalloc2 [1], which we're actively working to integrate into FF (bug 580408, but most of the action atm is on the jemalloc mailing list).

jemalloc2 has something called a heap profiler, and there's something about red zones in the code.  But I haven't dug into exactly what it does.

[1] http://www.canonware.com/jemalloc/ -- see the dev branch in git; master is old.

Comment 3

[Benoit Girard (:BenWa)]

(In reply to Ehsan Akhgari [:ehsan] from comment #1)
> Seems like Address Sanitizer already takes care of the first feature above
> (on Linux and OS X, of course) and more.  Do you think it's useful to have
> our own version of this functionality as well?

The goal I'm aiming for isn't to replace existing tools, but to provide an add-on that randomly audit memory and will catch some errors. If it doesn't get in the way of users people will be willing to run it 24/7.

(In reply to Justin Lebar [:jlebar] from comment #2)
> > 1) Electric fence styles feature to detect out of bounds read/write, usage after free: 
> > http://en.wikipedia.org/wiki/Electric_Fence
> 
> I wouldn't necessarily call this "heap profiling".
> 
> In any case, you should look at jemalloc2 [1], which we're actively working
> to integrate into FF (bug 580408, but most of the action atm is on the
> jemalloc mailing list).
> 
> jemalloc2 has something called a heap profiler, and there's something about
> red zones in the code.  But I haven't dug into exactly what it does.
> 
> [1] http://www.canonware.com/jemalloc/ -- see the dev branch in git; master
> is old.

I'll take a look. I must rather not have to reinvent this code here.

Comment 4

[Nick Fitzgerald [:fitzgen] [⏰PST; UTC-8]]

Note that :jimb and I are working on memory tooling for devtools. It is focused on the POV from JS, but will expose everything in the heap.

Follow along with the bugs in this graph: https://bugzilla.mozilla.org/showdependencygraph.cgi?id=960671

Comment 5

[Benoit Girard (:BenWa)]

robcee mentioned this to me the other day. I'm going to hold off on this and any similarly features since it should meet our needs.

Comment 6

[Paul Bone [:pbone]]

(In reply to Benoit Girard (:BenWa) from comment #0)
> 2) Liveness: We randomly choose to track a block being allocated in
> jemalloc. Tracked block get no read/write permissions. When they segfault we
> set them as read/write and record the last use time. Every minute (or the
> desired resolution) we reset to no read/write on the page giving us a rough
> estimate within a minute when the page was last used. We can unwind the
> allocation/last usage point/dealloc. With this info we can find memory we're
> leaking, or holding on for far too long in caches.

That is a really hard problem.  If memory is touched then it's live and that's probably okay.  But it might be possible to still reduce an application's memory usage.  If memory isn't touched then it might be dead, but still reachable, that could be a leak or it could just be that it might be used in the future.

Memory attribution may also be interesting, but more intensive for a profiler to examine.  If you can find paths through the memory graph that are holding a lot of memory reachable, and annotate those fields with how much memory they refer to.  The difficult thing here is that multiple references may lead to the same memory.  So A and B might both refer to 500MB each, but that doesn't mean your heap is 1000MB in total.  I think this is still worth considering, it's a feature that I've heard JS developers are interested in (anecdotal).