Last Comment Bug 695348 - (ZombieHunter) create a tool to analyze why compartments are alive
(ZombieHunter)
: create a tool to analyze why compartments are alive
Status: RESOLVED WONTFIX
[MemShrink:P2]
:
Product: Core
Classification: Components
Component: JavaScript Engine (show other bugs)
: Trunk
: All All
: -- normal with 11 votes (vote)
: ---
Assigned To: general
:
Mentors:
Depends on: 688225 680482 696174 740524 765873
Blocks: 723783 MemShrinkTools ZombieCompartments 691537
  Show dependency treegraph
 
Reported: 2011-10-18 09:04 PDT by Andrew McCreight (PTO-ish through 6-29) [:mccr8]
Modified: 2013-10-28 11:11 PDT (History)
30 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
print out the compartment of each object we dump (1022 bytes, patch)
2011-10-19 16:18 PDT, Andrew McCreight (PTO-ish through 6-29) [:mccr8]
no flags Details | Diff | Review
compartmentify JS dump heap output (2.57 KB, patch)
2011-10-22 12:10 PDT, Andrew McCreight (PTO-ish through 6-29) [:mccr8]
no flags Details | Diff | Review

Description Andrew McCreight (PTO-ish through 6-29) [:mccr8] 2011-10-18 09:04:28 PDT
The most common cause of zombie compartments (compartments that are live after their corresponding tabs are closed) at this point is addons (see Bug 691102, Bug 669730, Bug 672111, Bug 674535, etc.).  I think it would be useful to have a tool that could help the authors of such addons figure out what is causing the zombie compartments.  These users are sophisticated enough to write JS, but probably don't want to deal with hacking or debugging Firefox directly.

With a combination of cycle collector dumping and JS heap dumping (bug 680482), I feel like we should be able to explain why each compartment is alive, by giving a path from whatever kind of root to the compartment itself.  This should hopefully really just be a subset of existing data, so this is really just a matter of presenting information we have access to in a nicer fashion.

The first milestone for this would be a way to take a stock debug build of Firefox, set a few preferences or environment variables that will cause Firefox to dump out some kind of text log.  This log will then be processable by a simple command line script that explains why one or more compartments are still alive.

Longer term goals (to be addressed in follow up bugs) would be things like getting this to work in a release build (the main barrier there is probably getting JS heap dumping to work in a non-debug build), and entirely within the browser itself without any intermediary text logs (this would require a listener interface to JS heap dumping akin to the listener for CC dumping, then basically porting the analysis scripts from Python to JS).

I've been thinking that doing this kind of heap analysis within the browser itself would be a nice goal (you can basically do this already in Chrome), and zombie hunting seems like a nice well-defined problem to attack with this kind of tooling.
Comment 1 Andrew McCreight (PTO-ish through 6-29) [:mccr8] 2011-10-19 16:18:31 PDT
Created attachment 568250 [details] [diff] [review]
print out the compartment of each object we dump

This simple patch dumps out the compartment of each object dumped by the GC heap dumper.  Probably not how we want to do this for real, but it will give me data to mess around with analysis scripts.

I already have a script that can spit out which things in a compartment are directly rooted and how, by a few small tweaks to my JS dumping patch.  Will also need that kind of data for compartment->compartment edges.
Comment 2 Andrew McCreight (PTO-ish through 6-29) [:mccr8] 2011-10-22 12:10:14 PDT
Created attachment 568897 [details] [diff] [review]
compartmentify JS dump heap output

This prints out the name of every compartment, in addition to the compartment of every object.

I added a new script comp_find.py that understands the new format.  Given a GC log file and a compartment, it prints out every root that contains an object in that compartment, as well as every object in another compartment that points to the selected compartment.  It also prints out what is rooting those cross-compartment objects.

https://github.com/amccreight/heapgraph/tree/master/gc

I still need to put something together that will figure out what in other compartments is holding onto the DOM that is holding onto the current compartment via gray roots.
Comment 3 Andrew McCreight (PTO-ish through 6-29) [:mccr8] 2011-10-23 11:46:49 PDT
Steve Fink has a good post about how he tracked down some info about a zombie compartment using GDB: http://blog.mozilla.com/sfink/2011/08/02/zombie-hunting/
This is the sort of process I'd like to make more automatic, or at least make not involve GDB.
Comment 4 Jan Honza Odvarko [:Honza] 2011-12-02 05:07:20 PST
This sounds like a very promising tool for hunting mem leaks in Firebug (and other extensions)!

Any estimate when it could ready to use?

Also, it could be useful if it's possible to hook cycle collector heap dump (https://wiki.mozilla.org/Performance:Leak_Tools#JavaScript_heap_dump) from Javascript. This way extensions could get all the logs and make custom analysis dynamically (at the same time). At the present, I need to use Andrew's tools for analyzing heap logs: https://github.com/amccreight/heapgraph/tree/master/ manually, which is time consuming.

Should I report a new issue for it?

Honza
Comment 5 Andrew McCreight (PTO-ish through 6-29) [:mccr8] 2011-12-02 07:20:10 PST
I haven't really had time to work on it recently, unfortunately.

Yes, that is my long term plan.  The first step, in this bug, is to get it working with log dumping and external analysis scripts.  The second step is to figure out how to let JS get a representation of the CC graph, and port the analysis scripts from Python to JS.
Comment 6 Jan Honza Odvarko [:Honza] 2011-12-09 04:15:27 PST
Could we raise the priority for API that would allow JavaScript to dynamically analyze CC heap? This would allow extensions to create:

1) Custom memory analytic tools
2) Dynamic memory analysis
3) Memory related tests

This would be GREAT help for Firebug team and also other extensions authors when hunting down memory leaks.

Honza
Comment 7 Justin Lebar (not reading bugmail) 2011-12-10 02:54:03 PST
This is a priority for us, but Andrew has been off putting out fires lately.  We may yet take a different approach, e.g. bug 695480.
Comment 8 Andrew McCreight (PTO-ish through 6-29) [:mccr8] 2011-12-10 06:22:44 PST
(In reply to Jan Honza Odvarko from comment #6)
> Could we raise the priority for API that would allow JavaScript to
> dynamically analyze CC heap? This would allow extensions to create:
> 
> 1) Custom memory analytic tools
> 2) Dynamic memory analysis
> 3) Memory related tests

You should kind of be able to do that even without logs being exposed to JS.  I guess correlating a CC dump with a particular point in a test suite would be tricky.

I was talking to Luke, and it sounds like we should sometimes be able to extract information about where an object was allocated from type inference data.  I wonder if that would make logs more useful.

Also, as Justin says, bug 695480 has the potential to fix these kinds of zombie compartments by cleaving chrome to content connections, but I'm not sure of the timeline there.
Comment 9 Jan Honza Odvarko [:Honza] 2011-12-12 01:58:17 PST
(In reply to Andrew McCreight [:mccr8] from comment #8)
> (In reply to Jan Honza Odvarko from comment #6)
> You should kind of be able to do that even without logs being exposed to JS.
> I guess correlating a CC dump with a particular point in a test suite would
> be tricky.
The thing I wanted to do is:
1) Run Firebug tests suite (every test loads a page opens Firebug, checks the UI, closes the page)
2) Force CC collector to run
3) Use the CC heap dump JS API (doesn't exists now) to search the heap/object graph
and make sure the previous test page is not in the memory
4) If no -> all OK, if yes -> the test is creating a zombie compartment.

It would be slow and so, off by default (just for memory related testing)
Does it make sense?

> I was talking to Luke, and it sounds like we should sometimes be able to
> extract information about where an object was allocated from type inference
> data.  I wonder if that would make logs more useful.
Yes, it doesn't have to be so useful as it seems to be. But the biggest problem with the current logs is to identify the objects in the log and this is at least something, which could help in some cases.

> Also, as Justin says, bug 695480 has the potential to fix these kinds of
> zombie compartments by cleaving chrome to content connections, but I'm not
> sure of the timeline there.
Sounds good and promising. This could also help Firebug since 
chrome->wrapper->content->window scenario it typical (this is exactly what happens in Firebug).

Honza
Comment 10 Justin Lebar (not reading bugmail) 2011-12-12 04:10:11 PST
> 4) If no -> all OK, if yes -> the test is creating a zombie compartment.

So you just want to know *if* there's a zombie, not *why*?  That's much easier.  You can do that right now by doing what about:memory does: Force a GC/CC, then read the compartment memory reporters.

See

http://hg.mozilla.org/mozilla-central/file/bcc015450e7a/toolkit/components/aboutmemory/content/aboutMemory.js#l176

http://hg.mozilla.org/mozilla-central/file/bcc015450e7a/toolkit/components/aboutmemory/content/aboutMemory.js#l243

There's no easy way at the moment to get just the compartment memory reporters, but if you read all the multi-reporters, you should be able to pick out the compartment ones easily.
Comment 11 Jan Honza Odvarko [:Honza] 2011-12-12 04:18:48 PST
(In reply to Justin Lebar [:jlebar] from comment #10)
> > 4) If no -> all OK, if yes -> the test is creating a zombie compartment.
> 
> So you just want to know *if* there's a zombie, not *why*?  That's much
> easier.  You can do that right now by doing what about:memory does: Force a
> GC/CC, then read the compartment memory reporters.
I see, thanks for pointing this out.

Just to make sure, this part would be useful for *automatic* detection of zombies. But knowing *why* is, of course, still important in order to fix the underlying problem.

Honza
Comment 12 Jan Honza Odvarko [:Honza] 2011-12-21 09:36:46 PST
I spent a lot of time hunting memory leaks in Firebug recently (not much progress) and just wanted to highlight how useful it would be to have more info available in the CC heap dump and/or any other tools that help to find what is causing mem leaks/zombie compartments.

It would be invaluable not only for Firebug dev folks, but to all extension developers.

Honza
Comment 13 Andrew McCreight (PTO-ish through 6-29) [:mccr8] 2011-12-21 09:46:37 PST
What sort of information would be useful?
Comment 14 Andrew McCreight (PTO-ish through 6-29) [:mccr8] 2011-12-21 11:12:46 PST
Okay, so you should be able to figure out which test is creating a zombie compartment already, as Justin said in bug 10.  I guess though once you have a reproducable test case that causes a zombie compartment, you'd like better information about why it is being held alive?
Comment 15 Jan Honza Odvarko [:Honza] 2011-12-21 23:43:13 PST
(In reply to Andrew McCreight [:mccr8] from comment #14)
> Okay, so you should be able to figure out which test is creating a zombie
> compartment already, as Justin said in bug 10.
Yes

> I guess though once you have
> a reproducable test case that causes a zombie compartment, you'd like better
> information about why it is being held alive?
Precisely

I am using the find_roots.py - it's great, but I just don't know what are the objects in the chain.

So, the biggest problem is to identify the objects in the log. I agree that line-number and file-url don't have to be that useful, but in the end - it's at least something. Better could be if it's possible to see the 'shape' of the object (all property values).

Of course, the best would be to have an object ID that I can directly match against the current Firebug structure and dynamically find the right object.

Also, having API - exposed to JS - that can be used to analyze the heap dynamically is huge. This + more info to identify the object could encourage folks to create new tools/extensions.

And it would be much simpler (from UX perspective) to get the object chain zombie -> root - it could be just one click (from within Firefox). Currently I need to do several manual steps.

Honza
Comment 16 Andrew McCreight (PTO-ish through 6-29) [:mccr8] 2012-01-26 07:08:47 PST
My hope is that compartment-per-global will make this easier to do.
Comment 17 Chris Pickett 2012-03-29 11:46:03 PDT
I extracted the suggestion as to how one might automatically find zombies in comment #10 into Bug 740524.
Comment 18 Nicholas Nethercote [:njn] (on vacation until July 11) 2012-07-05 23:36:24 PDT
(In reply to Andrew McCreight [:mccr8] from comment #16)
> My hope is that compartment-per-global will make this easier to do.

I heard that CPG landed.  Any news on this front? :)
Comment 19 Andrew McCreight (PTO-ish through 6-29) [:mccr8] 2012-07-06 08:10:35 PDT
(In reply to Nicholas Nethercote [:njn] from comment #18)
> I heard that CPG landed.  Any news on this front? :)

Unfortunately I'm too busy fixing CPG regressions to work on this. ;)  Is this still important given that we have many less zombie compartments?  I guess it could be useful for analyzing things like the Google Reader leak.
Comment 20 Kyle Huey [:khuey] (khuey@mozilla.com) 2012-07-06 08:13:41 PDT
I still think we want this, for web devs if nothing else.
Comment 21 Nicholas Nethercote [:njn] (on vacation until July 11) 2012-08-29 20:58:20 PDT
Does this still need to be MemShrink:P1?
Comment 22 Andrew McCreight (PTO-ish through 6-29) [:mccr8] 2012-08-29 21:36:47 PDT
Nah.
Comment 23 Andrew McCreight (PTO-ish through 6-29) [:mccr8] 2013-10-10 16:41:00 PDT
about:compartments has been retired, and this bug probably can be, too.  Fine-grained leak hunting tools are still of interest.

Note You need to log in before you can comment on or make changes to this bug.