50338 - instrument boehm GC to dump "in use" information

Status: RESOLVED INCOMPLETE

(Core :: XPCOM, defect, P3)

Description

[Chris Waterson]

We may be able to get more traction on the bloat problem if we had a tool that
could "snapshot" the current memory state, and identify all the objects that
were currently in use. (Ideally, with information about who allocated each
object.) It was suggested that the Boehm collector could probably do this.

Comment 1

[Brendan Eich [:brendan]]

Attachment: trivial extension to nsTraceMalloc

Comment 2

[Brendan Eich [:brendan]]

This patch provides a function, NS_TraceMallocDumpAllocation(const char
*pathname), that can be called from anywhere in the code.  I'll add a patch to
provide a JS native function TraceMallocDumpAllocations in a sec.  You have to
--enable-trace-malloc in your Linux build, as usual, and you should run mozilla
with --trace-malloc /dev/null or a real file, to capture allocations and their
compressed stack backtraces.

The output file is likely to be very large; beware.  Obviously, we could do much
better by minimizing callsite tree path redundancy, as the trace-malloc logfile
format strives to do in a binary fashion.

/be

Comment 3

[Brendan Eich [:brendan]]

Attachment: Add TraceMallocDumpAllocations JS function

Comment 4

[Brendan Eich [:brendan]]

Attachment: consolidated patch with a few cleanups

Comment 5

[Brendan Eich [:brendan]]

Attachment: Clean up header line spacing to match columns; add semicolon separator between callsites for better perl/sed/awk munging

Comment 6

[Brendan Eich [:brendan]]

Attachment: Check NS_TraceMallocDumpAllocations' return value and report JS error if necessary

Comment 7

[Brendan Eich [:brendan]]

Starting ./mozilla --trace-malloc /dev/null to first window loading mozilla.org,
then loading file:/tmp/dump.html (see next attachment) results in a 250MB file
with > 126,000 lines, one for each live allocation when I clicked on the Dump
button in the form after entering /tmp/big.dump in the text input.

Still, it's a crude attempt at what this bug wants, and it can be post-processed
by good ol' Unix utilities.  Go nuts.

/be

Comment 8

[Chris Waterson]

Attachment: "web page" that will dump live allocations to /tmp/dump.log

Comment 9

[Chris Waterson]

Attachment: blame.pl, which will process dump.log and attribute allocations by call stack

Comment 10

[Chris Waterson]

Attachment: a style sheet to make it pretty

Comment 11

[Chris Waterson]

Put a log file of "opening a new window" at

  http://ftp.mozilla.org/bloat-reports/2000-08-28-live-objects.html

Take a look (may take a while for the FTP server to sync up). It's pretty cool.
Some first order comments...

  - About 9Mb of live allocations in the heap
  - NS_InitXPCOM allocates 300Kb.
  - ~170KB is from PR_NewLock().
  - ~500KB comes from 300 XUL templates. How on earth did we get
    that many XUL templates?
  - 1.2Mb is from JS_Malloc().
  - At least 90Kb of parser junk lying around. I thought parsers
    got killed as soon as they were done being used?
  - 1.2Mb of string data lying around
  - My global history is taking up about 80Kb. AFAICT, I only have
    a dozen or so sites. We should see how this scales...
  - XBL accounts for ~400Kb.

N.B. that I've done nothing to deal with recursive routines, so I think that
this'll be double-counting a routine that's entered >1 time on a call stack.

Brendan: how soon until you'd feel comfortable checking these patches in? When
you do so, I'll check my scripts and stuff into mozilla/tools somewhere.

Comment 12

[Chris Waterson]

Oops, make that

  http://ftp.mozilla.org/pub/bloat-reports/2000-08-28-live-objects.html

Still not there. Should be there in an hour or so.

Comment 13

[Brendan Eich [:brendan]]

Attachment: sorry, forgot to attach my (html3.2-ish) dump.html form page

Comment 14

[Chris Waterson]

Checked in blame.pl and blame.css into mozilla/tools/trace-malloc, a new
directory I've created to house brendan's family of trace-malloc based tools.
Brendan, check in your TraceMallocDumpAllocations() patch. I crave it.

Comment 15

[Brendan Eich [:brendan]]

I checked in yesterday afternoon.

/be

Comment 16

[Brendan Eich [:brendan]]

http://ftp.mozilla.org/pub/bloat-reports/2000-08-28-live-object.html is the 
right URL, it seems.  Good stuff, mysteries of the deep.

/be

Comment 17

[chris hofmann]

over to thesteve

Comment 18

[Benjamin Smedberg]

boehmgc is no longer in our tree