668695 - analysis scripts for cycle collector log format

Status: NEW

(Core :: Cycle Collector, enhancement)

Description

[Andrew McCreight [:mccr8]]

+++ This bug was initially created as a clone of Bug #649532 +++

Python scripts to analyze the new cycle collector logs.

- parsing library
- root finding
- cycle collector checker

I'm making a separate bug for the scripts themselves.

Comment 1

[Andrew McCreight [:mccr8]]

Attachment: Python scripts for CC logs: root finding, checking CC results

Here are the scripts.  parse_cc_graph.py is a library that is a common log parser.  Unfortunately, parsing files is a bit slower than I'd like. It takes about 6 seconds to parse a 40 meg log file.  Well, I don't know if that is good or bad.

Unlike peter's original scripts, I don't have a preprocessing step in the root finder to the reverse the graph, because reversing the graph doesn't seem to take that long compared to parsing it in.  I could also add a script that reverses the graph, as that could be useful for manual exploration of the graph.

Comment 2

[Peter Van der Beken [:peterv]]

argparse seems to require Python 2.7, could we use optparse for now (even though it's deprecated)?

For the root finding script I think it's helpful to know how many missing edges there are for a "root", my script printed that out. When looking at a refcount tree you actually know how many missing refs you're looking for. It's very rare but I've had cases where the refcount of a root decreased between the final CC and shutdown, so in that case I knew that the refcount tree wasn't showing the exact state from when the final CC happened.

Very minor, but I get distracted by edges that don't have a description, because they look different:

    --[xpc_GetJSPrivate(obj)]-> 0x1269b5070 [XPCWrappedNative (XULDocument)]
    --[mIdentity]-> 0x1009dda00 [nsDocument (XUL) places.xul]
    --[mCommandDispatcher]-> 0x12699fb60 [nsXULCommandDispatcher]
    ---> 0x1269e0190 [nsGenericElement (XUL) commandset]
    --[GetParent()]-> 0x1269cdba0 [nsGenericElement (XUL) commandset]
    --[GetParent()]-> 0x11d0e36d0 [nsGenericElement (XUL) commandset]
    --[GetParent()]-> 0x11d0cc200 [nsGenericElement (XUL) window]

I'd leave the (empty) square brackets in that case.

Speed of at least the root finder seemed ok to me.

Comment 3

[Andrew McCreight [:mccr8]]

(In reply to Peter Van der Beken [:peterv] from comment #2)
> argparse seems to require Python 2.7, could we use optparse for now (even
> though it's deprecated)?

Sure.  I'm not really a Python expert, so I just went with whatever Google turned up.

> For the root finding script I think it's helpful to know how many missing
> edges there are for a "root", my script printed that out. When looking at a
> refcount tree you actually know how many missing refs you're looking for.
> It's very rare but I've had cases where the refcount of a root decreased
> between the final CC and shutdown, so in that case I knew that the refcount
> tree wasn't showing the exact state from when the final CC happened.

Oh, good point.  I didn't realize I'd missed that.

> Very minor, but I get distracted by edges that don't have a description,
> because they look different:

> I'd leave the (empty) square brackets in that case.

I'll fix that.
 
> Speed of at least the root finder seemed ok to me.

Okay, that's good.  I did some profiling and found that about half the time of the script was spent storing the edge names, which was a bit sad.

I have some slightly newer versions of these scripts at https://github.com/amccreight/heapgraph/tree/master/cc but I don't think I changed anything substantial in there.  They could actually be the same.  Or slightly older, come to think of it... I'll compare and post here if there are any newer changes there.

Comment 4

[Andrew McCreight [:mccr8]]

I added support for unknown edges so it looks like the following:

0x14707e6d0 [nsGenericElement (xhtml) ul]
    --[mAttrsAndChildren[i]]-> 0x146daeba0 [nsGenericDOMDataNode]

    Root 0x14707e6d0 is a ref counted object with 1 unknown edge(s).
    known edges:
       0x14707eb50 [nsGenericElement (xhtml) li] --[GetParent()]-> 0x14707e6d0
       0x146daefb0 [nsGenericElement (xhtml) li] --[GetParent()]-> 0x14707e6d0

I also changed empty edges to be displayed with the brackets, and likewise with nodes.  Before my fix nodes with empty names would make the script throw an exception.  Oops!

Still need to change to the old system of arguments.

Comment 5

[Andrew McCreight [:mccr8]]

Attachment: Python scripts for CC logs: root finding, checking CC results

In addition to the changes in Comment #4, I changed both scripts to use optparse, and updated the comments in the CC checker a little.  I also synced this up to github.  Probably easier to read these scripts there: https://github.com/amccreight/heapgraph/tree/master/cc

Comment 6

[Andrew McCreight [:mccr8]]

I was also wondering if maybe having a reference to the github repo might be useful, for the latest version.  My plan is to tag whatever versions get committed to m-c.

Comment 7

[Andrew McCreight [:mccr8]]

One of the nice things about having a log parsing library is that it is easy to write new scripts.  It only took me a few minutes to whip up this little script to analyze the classes of garbage objects:

https://github.com/amccreight/heapgraph/blob/master/cc/garbage_census.py

I did this for bug 682219, where there's some kind of leak with a weird setting, and the CC isn't collecting anything, but if you do a CC log dump (which does a WANT_ALL_TRACES) suddenly the CC finds things!  It is a little unsettling.  Anyways, with this script I was able to see that there's a ton of XUL stuff that is collected.  I don't know if it is actually useful.

Comment 8

[Andrew McCreight [:mccr8]]

I still need to land this in the tree one of these days...