695348 - (ZombieHunter) create a tool to analyze why compartments are alive

Status: RESOLVED WONTFIX

(Core :: JavaScript Engine, defect)

Description

[Andrew McCreight [:mccr8]]

The most common cause of zombie compartments (compartments that are live after their corresponding tabs are closed) at this point is addons (see Bug 691102, Bug 669730, Bug 672111, Bug 674535, etc.).  I think it would be useful to have a tool that could help the authors of such addons figure out what is causing the zombie compartments.  These users are sophisticated enough to write JS, but probably don't want to deal with hacking or debugging Firefox directly.

With a combination of cycle collector dumping and JS heap dumping (bug 680482), I feel like we should be able to explain why each compartment is alive, by giving a path from whatever kind of root to the compartment itself.  This should hopefully really just be a subset of existing data, so this is really just a matter of presenting information we have access to in a nicer fashion.

The first milestone for this would be a way to take a stock debug build of Firefox, set a few preferences or environment variables that will cause Firefox to dump out some kind of text log.  This log will then be processable by a simple command line script that explains why one or more compartments are still alive.

Longer term goals (to be addressed in follow up bugs) would be things like getting this to work in a release build (the main barrier there is probably getting JS heap dumping to work in a non-debug build), and entirely within the browser itself without any intermediary text logs (this would require a listener interface to JS heap dumping akin to the listener for CC dumping, then basically porting the analysis scripts from Python to JS).

I've been thinking that doing this kind of heap analysis within the browser itself would be a nice goal (you can basically do this already in Chrome), and zombie hunting seems like a nice well-defined problem to attack with this kind of tooling.

Comment 1

[Andrew McCreight [:mccr8]]

Attachment: print out the compartment of each object we dump

This simple patch dumps out the compartment of each object dumped by the GC heap dumper.  Probably not how we want to do this for real, but it will give me data to mess around with analysis scripts.

I already have a script that can spit out which things in a compartment are directly rooted and how, by a few small tweaks to my JS dumping patch.  Will also need that kind of data for compartment->compartment edges.

Comment 2

[Andrew McCreight [:mccr8]]

Attachment: compartmentify JS dump heap output

This prints out the name of every compartment, in addition to the compartment of every object.

I added a new script comp_find.py that understands the new format.  Given a GC log file and a compartment, it prints out every root that contains an object in that compartment, as well as every object in another compartment that points to the selected compartment.  It also prints out what is rooting those cross-compartment objects.

https://github.com/amccreight/heapgraph/tree/master/gc

I still need to put something together that will figure out what in other compartments is holding onto the DOM that is holding onto the current compartment via gray roots.

Comment 3

[Andrew McCreight [:mccr8]]

Steve Fink has a good post about how he tracked down some info about a zombie compartment using GDB: http://blog.mozilla.com/sfink/2011/08/02/zombie-hunting/
This is the sort of process I'd like to make more automatic, or at least make not involve GDB.

Comment 4

[Jan Honza Odvarko [:Honza] (always need-info? me)]

This sounds like a very promising tool for hunting mem leaks in Firebug (and other extensions)!

Any estimate when it could ready to use?

Also, it could be useful if it's possible to hook cycle collector heap dump (https://wiki.mozilla.org/Performance:Leak_Tools#JavaScript_heap_dump) from Javascript. This way extensions could get all the logs and make custom analysis dynamically (at the same time). At the present, I need to use Andrew's tools for analyzing heap logs: https://github.com/amccreight/heapgraph/tree/master/ manually, which is time consuming.

Should I report a new issue for it?

Honza

Comment 5

[Andrew McCreight [:mccr8]]

I haven't really had time to work on it recently, unfortunately.

Yes, that is my long term plan.  The first step, in this bug, is to get it working with log dumping and external analysis scripts.  The second step is to figure out how to let JS get a representation of the CC graph, and port the analysis scripts from Python to JS.

Comment 6

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Could we raise the priority for API that would allow JavaScript to dynamically analyze CC heap? This would allow extensions to create:

1) Custom memory analytic tools
2) Dynamic memory analysis
3) Memory related tests

This would be GREAT help for Firebug team and also other extensions authors when hunting down memory leaks.

Honza

Comment 7

[Justin Lebar (not reading bugmail)]

This is a priority for us, but Andrew has been off putting out fires lately.  We may yet take a different approach, e.g. bug 695480.

Comment 8

[Andrew McCreight [:mccr8]]

(In reply to Jan Honza Odvarko from comment #6)
> Could we raise the priority for API that would allow JavaScript to
> dynamically analyze CC heap? This would allow extensions to create:
> 
> 1) Custom memory analytic tools
> 2) Dynamic memory analysis
> 3) Memory related tests

You should kind of be able to do that even without logs being exposed to JS.  I guess correlating a CC dump with a particular point in a test suite would be tricky.

I was talking to Luke, and it sounds like we should sometimes be able to extract information about where an object was allocated from type inference data.  I wonder if that would make logs more useful.

Also, as Justin says, bug 695480 has the potential to fix these kinds of zombie compartments by cleaving chrome to content connections, but I'm not sure of the timeline there.

Comment 9

[Jan Honza Odvarko [:Honza] (always need-info? me)]

(In reply to Andrew McCreight [:mccr8] from comment #8)
> (In reply to Jan Honza Odvarko from comment #6)
> You should kind of be able to do that even without logs being exposed to JS.
> I guess correlating a CC dump with a particular point in a test suite would
> be tricky.
The thing I wanted to do is:
1) Run Firebug tests suite (every test loads a page opens Firebug, checks the UI, closes the page)
2) Force CC collector to run
3) Use the CC heap dump JS API (doesn't exists now) to search the heap/object graph
and make sure the previous test page is not in the memory
4) If no -> all OK, if yes -> the test is creating a zombie compartment.

It would be slow and so, off by default (just for memory related testing)
Does it make sense?

> I was talking to Luke, and it sounds like we should sometimes be able to
> extract information about where an object was allocated from type inference
> data.  I wonder if that would make logs more useful.
Yes, it doesn't have to be so useful as it seems to be. But the biggest problem with the current logs is to identify the objects in the log and this is at least something, which could help in some cases.

> Also, as Justin says, bug 695480 has the potential to fix these kinds of
> zombie compartments by cleaving chrome to content connections, but I'm not
> sure of the timeline there.
Sounds good and promising. This could also help Firebug since 
chrome->wrapper->content->window scenario it typical (this is exactly what happens in Firebug).

Honza

Comment 10

[Justin Lebar (not reading bugmail)]

> 4) If no -> all OK, if yes -> the test is creating a zombie compartment.

So you just want to know *if* there's a zombie, not *why*?  That's much easier.  You can do that right now by doing what about:memory does: Force a GC/CC, then read the compartment memory reporters.

See

http://hg.mozilla.org/mozilla-central/file/bcc015450e7a/toolkit/components/aboutmemory/content/aboutMemory.js#l176

http://hg.mozilla.org/mozilla-central/file/bcc015450e7a/toolkit/components/aboutmemory/content/aboutMemory.js#l243

There's no easy way at the moment to get just the compartment memory reporters, but if you read all the multi-reporters, you should be able to pick out the compartment ones easily.

Comment 11

[Jan Honza Odvarko [:Honza] (always need-info? me)]

(In reply to Justin Lebar [:jlebar] from comment #10)
> > 4) If no -> all OK, if yes -> the test is creating a zombie compartment.
> 
> So you just want to know *if* there's a zombie, not *why*?  That's much
> easier.  You can do that right now by doing what about:memory does: Force a
> GC/CC, then read the compartment memory reporters.
I see, thanks for pointing this out.

Just to make sure, this part would be useful for *automatic* detection of zombies. But knowing *why* is, of course, still important in order to fix the underlying problem.

Honza

Comment 12

[Jan Honza Odvarko [:Honza] (always need-info? me)]

I spent a lot of time hunting memory leaks in Firebug recently (not much progress) and just wanted to highlight how useful it would be to have more info available in the CC heap dump and/or any other tools that help to find what is causing mem leaks/zombie compartments.

It would be invaluable not only for Firebug dev folks, but to all extension developers.

Honza

Comment 13

[Andrew McCreight [:mccr8]]

What sort of information would be useful?

Comment 14

[Andrew McCreight [:mccr8]]

Okay, so you should be able to figure out which test is creating a zombie compartment already, as Justin said in bug 10.  I guess though once you have a reproducable test case that causes a zombie compartment, you'd like better information about why it is being held alive?

Comment 15

[Jan Honza Odvarko [:Honza] (always need-info? me)]

(In reply to Andrew McCreight [:mccr8] from comment #14)
> Okay, so you should be able to figure out which test is creating a zombie
> compartment already, as Justin said in bug 10.
Yes

> I guess though once you have
> a reproducable test case that causes a zombie compartment, you'd like better
> information about why it is being held alive?
Precisely

I am using the find_roots.py - it's great, but I just don't know what are the objects in the chain.

So, the biggest problem is to identify the objects in the log. I agree that line-number and file-url don't have to be that useful, but in the end - it's at least something. Better could be if it's possible to see the 'shape' of the object (all property values).

Of course, the best would be to have an object ID that I can directly match against the current Firebug structure and dynamically find the right object.

Also, having API - exposed to JS - that can be used to analyze the heap dynamically is huge. This + more info to identify the object could encourage folks to create new tools/extensions.

And it would be much simpler (from UX perspective) to get the object chain zombie -> root - it could be just one click (from within Firefox). Currently I need to do several manual steps.

Honza

Comment 16

[Andrew McCreight [:mccr8]]

My hope is that compartment-per-global will make this easier to do.

Comment 17

[Chris Pickett]

I extracted the suggestion as to how one might automatically find zombies in comment #10 into Bug 740524.

Comment 18

[Nicholas Nethercote [inactive]]

(In reply to Andrew McCreight [:mccr8] from comment #16)
> My hope is that compartment-per-global will make this easier to do.

I heard that CPG landed.  Any news on this front? :)

Comment 19

[Andrew McCreight [:mccr8]]

(In reply to Nicholas Nethercote [:njn] from comment #18)
> I heard that CPG landed.  Any news on this front? :)

Unfortunately I'm too busy fixing CPG regressions to work on this. ;)  Is this still important given that we have many less zombie compartments?  I guess it could be useful for analyzing things like the Google Reader leak.

Comment 20

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I still think we want this, for web devs if nothing else.

Comment 21

[Nicholas Nethercote [inactive]]

Does this still need to be MemShrink:P1?

Comment 22

[Andrew McCreight [:mccr8]]

Nah.

Comment 23

[Andrew McCreight [:mccr8]]

about:compartments has been retired, and this bug probably can be, too.  Fine-grained leak hunting tools are still of interest.