`leaked 1 window(s) until shutdown` is not actionable

NEW
Unassigned

Status

defect
4 years ago
4 years ago

People

(Reporter: Yoric, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

I have just spent the last 24h trying to find out what causes a leak in my windows, and I haven't found yet.

We need a tool that helps us find what references the windows.
Note: walking the results of CCAnalyzer can give some information. It's not much, but it's a start.
njn, ochameau, any idea how we could do that?
Flags: needinfo?(poirot.alex)
Flags: needinfo?(n.nethercote)
If you really have no idea what is causing the leak, you need to get the CCAnalyzer to produce a CC log, then run find_roots.py on it ( https://github.com/amccreight/heapgraph ). That will tell you what is entraining the window.
Flags: needinfo?(n.nethercote)
I believe that this should be built into mochitests.
I have nothing to add beyond what mccr8 said.
(In reply to David Rajchenbach-Teller [:Yoric] (away June 22 - July 7th, use "needinfo") from comment #4)
> I believe that this should be built into mochitests.

Yes, that would be handy.  It looks like this particular leak detector is the ++DOMWINDOW one.  The CCAnalyzer doesn't actually work with e10s right now. I should probably work on that again at some point. But extending it to print out why an object is alive would be additional work, basically porting the find_roots script to JS.
Haven't used your find_roots, but isn't that pretty much the same what about:cc's "Find Roots" is.
https://pastebin.mozilla.org/8837327 findRoots is rather trivial.
I'll try and see if I can find some time to work on it during Whistler.
Assignee: nobody → dteller
As the cycle detection takes place at the end of tests, when we don't have windows anymore, I believe that the best way to get something understandable is to output some dot code.
The interesting thing is usually the object that is holding alive the window (and the known references to it), not why that object is holding the window alive.
Indeed. Is that a difference between find_roots.py and about:cc or a reply to my idea of putting dot code.
Flags: needinfo?(continuation)
(In reply to David Rajchenbach-Teller [:Yoric] (away June 22 - July 7th, use "needinfo") from comment #11)
> Indeed. Is that a difference between find_roots.py and about:cc or a reply
> to my idea of putting dot code.

The dot code. Just showing a giant blob of everything in the CC graph won't be very useful, though maybe that's not what you meant.
Flags: needinfo?(continuation)
I was thinking of printing the references that hold the window, the references that hold these, etc. on N generations, with N to be determined and possibly configurable.
(In reply to David Rajchenbach-Teller [:Yoric] (away June 22 - July 7th, use "needinfo") from comment #13)
> I was thinking of printing the references that hold the window, the
> references that hold these, etc. on N generations, with N to be determined
> and possibly configurable.

The references to the known references to an object aren't very interesting. What you really need to figure out is the unknown references are, and listing the known ones mostly just lets you rule out things you know about (by code inspection or what have you).
I'm not sure what you mean by the "known" and "unknown" references.

Also, looking at the CC Logger, it seems that the most precise information I can get on JS references is the compartment in which they live (and perhaps the event listener, if there is one involved). That's not nearly sufficient if the reference lives e.g. in a map, or in a closure.

I guess I could try and walk the gc graph, but that's starting to look a bit overcomplex. What do you think, Andrew? Also, do you want to chat about it in Whistler?
Flags: needinfo?(continuation)
unknown reference in CC means an edge which isn't traversed.
It may mean for example necko->some cycle collectable object edge (necko stuff isn't cycle collectable), or cycle collectable object (1) -> another cycle collectable object (2) where
(1) isn't suspected (no addref/release has happened after the previous CC), so cycle collector doesn't know about it.
The CC graph should give you some information about what refers to the JS object, though you have to look at the entire graph to figure that out.  We don't right now have any facilities to pass over the GC graph to JS like we do with the CC.
Flags: needinfo?(continuation)
Yoric, Here is some tweaks I made to the CC code in order to print much more information when dumping the CC:
  https://github.com/ochameau/mozilla-central/commit/b5bd76b70cb3b64adaebbcb06e91ab4b985006a5#diff-a069514c3943aae86ea615b06262a967L507
This patch is most likely outdated, and also, it does much more than adding naive information about the nodes. It also display the allocation site for each object (costly operation, that requires setting an object metadata callback in every js scope).
Flags: needinfo?(poirot.alex)
Plenty of ideas discussed with ochameau, but no time to work on it atm.
Assignee: dteller → nobody
I may attempt some work here next time I am faced with debugging an issue like this, as my current technique resembles banging my head on a wall for several days. :)
FYI: here is a simple stupid way to check if a given object is leaked of not:

  let w = Cu.getWeakReference(object);
  require("sdk/timers").setInterval(function () {
    Cu.forceGC();
    console.log("is leaked?", !!w.get());
  }, 2000);

I'm using CCAnalyzer to read the GC graph.
Somehow find a way to load this as a CommonJS module (I put it in devtools folders...)
  http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/cc-analyzer.js

Then do that from browser console:
  let {CCAnalyzer} = require("devtools/toolkit/cc-analyzer");
  let cc = new CCAnalyzer();
  cc.run(()=>{ console.log("collected"); });

Once "collected" is logged, you can inspect `cc` and especially its `cc.graph` and `cc.edges`.
By default CCAnalyzer only returns the CC, which is a very limited view of JS objects.
But you can tweak it to log the whole GC, but it takes a while and allocate tons of new JS objects...
To do that add `this.listener = this.listener.allTraces();` in its run method.

Having said all that, I don't think we can do a magic algorithm to figure out leaks.
We might be able to do something really simple to expose cross compartment wrappers refering to the leaked window, but we wouldn't have much useful information to print about this object expect it's property names. What we could do is expose a CCAnalyser instance of the leak and let the developer read the CC/GC Graph out of it in order to try to guess which object it is.

What we could do is first make cc-analyzer be available in firefox builds and easily usable from browser console or something.
And then tweak it to easily fetch the GC and some helpers to easily filter graph and edges to identify cross crompartment wrappers or other typical scheme of leaks.
(In reply to Alexandre Poirot [:ochameau] from comment #21)
>   let w = Cu.getWeakReference(object);
>   require("sdk/timers").setInterval(function () {
>     Cu.forceGC();
>     console.log("is leaked?", !!w.get());
>   }, 2000);
Note, forceGC triggers only GC, so one might want to call also forceCC.

And isn't fitzgen adding some kind of object graph tool to devtools.
 

Btw, nsICycleCollectorListener can show a lot larger part of the JS world too if one uses wantAllTraces mode.
But that really may create huge graphs - it effectively disables all the dont-put-certainly-alive-objects-to-the-CC-graph optimizations.

And, GC/CC graphs don't really help with shutdown leaks. Sure you can use the information to see the known edges, but it is
the unknown one which one needs to find.

Runtime leaks (leaks which don't show up in XPCOM_MEM_LEAK_LOG) are easier to debug, at least in case of cycle collectable objects. First you create a CC log to get all the known edges to certain object. Then you add a breakpoint to object's
::Release() and then shutdown the browser (or tab or whatever is enough to release the object) to see all the edges, also the unknown ones. One just checks the stack trace of each ::Release() call. Then compare all the edges to the known ones, and one (or more) of the unknown edges is usually the one causing the leak.
You need to log in before you can comment on or make changes to this bug.