688072 - Windows Heap Profiling Tool

Status: RESOLVED INCOMPLETE

(Core Graveyard :: Tracking, defect)

Description

[Katelyn Gadd (:kael)]

This bug is about the windows heap profiler I built for the performance team:

The GitHub repository is located at https://github.com/kevingadd/HeapProfiler.

There are two blog posts describing its use at:
http://www.luminance.org/code/2011/08/01/a-journey-into-linker-hell-and-a-mistake
http://www.luminance.org/blog/code/2011/06/03/diving-into-memory-usage-with-heap-profiler

To compile it, you need a 2010 or newer version of the C# compiler (it uses language version 4 and the libraries that came with it). It uses two libraries that can also be found on my GitHub profile, Fracture and DataMangler - you can pull them in using included git externals that will put them in the correct location.

Current status of the profiler:
Tracks all allocations via the standard windows heap inside the program it's monitoring, and captures the first 32 stack frames associated with each allocation call. Snapshots of the current state of the application heap are taken periodically and saved.
Captures are stored in a simple key-value store format optimized for search. Symbol information for the application's modules is pulled in at the time of capture and cached in the KVS so that a capture can be viewed without having the symbols for the exact version of the application that was captured (note that source information will, of course, be incorrect if the source code has changed.)
Two points within the capture's timeline can be diffed to identify memory that was allocated and freed between those points in time. The diff can be categorized and searched by basic symbol information (function, module, folder, etc.)
You can view a very simple visualization of the heap at a given point in time and identify the function(s) involved in allocating the data at an address on the heap. This UI does not yet provide any of the searching or filtering features that would probably be useful - however, adding them should be relatively straightforward because the logic necessary is similar to that used by the diffing algorithm and the diff viewer UI.

Support for jemalloc will require either changes to jemalloc (vlad had a patch for this, but I never got it to work) or manual instrumentation of the functions we use to allocate.
Support for capturing JS stacks will require replacing use of Microsoft's kernel heap instrumentation with custom instrumentation inside Firefox. The xptrace library was intended to address this problem, along with jemalloc support, by allowing us to embed disabled-by-default probes into our jemalloc and malloc entry points, and then enable those probes and use them to capture both C++ and JS stacks at the point of each allocation.
Switching to custom instrumentation will also lift the 32 stack frame limit (that is imposed by the Windows kernel instrumentation) and as a result make the allocation data much more accurate, so that's another good reason to do it. It might be possible to leverage code from massif/valgrind for these purposes.

Comment 1

[BMO Automation]

Marking all tracking bugs which haven't been updated since 2014 as INCOMPLETE.
If this bug is still relevant, please reopen it and move it into a bugzilla component related to the work
being tracked. The Core: Tracking component will no longer be used.