I started investigating this and I have a few findings Tldr: - leak is linked to the content-process DebuggerServer that lives in the same process as the tab (where we already have a DebuggerServer) - leak is linked to a "scripts" map, held by BreakpointActors, which are never cleaned up We are connecting to several content processes during this test (5 in my local tests). The content process that makes this test leak is the content process of the tab targeted by the toolbox. If I skip this specific content process and don't connect to it, there is no leak. The actorIDs of the descriptors are relatively stable across runs, so I simply filtered by processDescriptorFront.actorID to verify this theory. In practice, we already have a DebuggerServer created in this process, because of Tab target of the toolbox. So when we connect to this process target, we create another DebuggerServer in the same process. The content-process.jsm script creates a dedicated loader, which means we really use separate DebuggerServers: https://searchfox.org/mozilla-central/source/devtools/server/startup/content-process.jsm#29-38 If we modify content-process.jsm to use the shared loader instead of creating a new one, the leak disappears as well. Finally, simply setting `invisibleToDebugger` to false is not enough to fix the leak. More interesting I also looked at the test itself to understand which step was leaking precisely. Turns out the test only leaks when you add a breakpoint via `await addBreakpoint(dbg, "asm.js", 7);`. Diving deeper into what leaks in the breakpoint creation, it seems related to a map held by the BreakpointActor: https://searchfox.org/mozilla-central/source/devtools/server/actors/breakpoint.js#40. This map is supposed to be cleared when destroying the actor, but breakpoint actors are never destroyed (at least not in this test). Modifying ThreadActor to destroy all breakpoint actors, fixes the leak for me locally. That being said, I don't understand why this turned into a leak here and not before. Even without enabling windowless service workers, the breakpoint actors are never destroyed. What is it about creating an additional DebuggerServer in the same process that makes this map leak? So far I don't know. I also verified that all the DebuggerServers were correctly destroyed, couldn't spot anything wrong there. I have two try pushes in progress: - call BreakpointActor destroy: https://treeherder.mozilla.org/#/jobs?repo=try&revision=43d5425cf262483d97ae80fa42e81308fdb456e8 - call BreakpointActor destroy + enable windowless service workers: https://treeherder.mozilla.org/#/jobs?repo=try&revision=24fc8c9d1adf9d57b7562ea369cac81e2708417b However if you look at the original patch that was backed out: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&revision=47f8d76aa4fc9a21a03d7163cc0b4747a65adfe0 a lot of failures are timeouts, and don't seem linked to this leak. So let's wait to see what try says, but I think more work might be needed to enable the feature (probably should file other bugs).
Bug 1609201 Comment 3 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
#### summary I started investigating this and I have a few findings Tldr: - leak is linked to the content-process DebuggerServer that lives in the same process as the tab (where we already have a DebuggerServer) - leak is linked to a "scripts" map, held by BreakpointActors, which are never cleaned up #### try pushes I have two try pushes in progress: - call BreakpointActor destroy: https://treeherder.mozilla.org/#/jobs?repo=try&revision=43d5425cf262483d97ae80fa42e81308fdb456e8 - call BreakpointActor destroy + enable windowless service workers: https://treeherder.mozilla.org/#/jobs?repo=try&revision=24fc8c9d1adf9d57b7562ea369cac81e2708417b However if you look at the original patch that was backed out: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&revision=47f8d76aa4fc9a21a03d7163cc0b4747a65adfe0 a lot of failures are timeouts, and don't seem linked to this leak. So let's wait to see what try says, but I think more work might be needed to enable the feature (probably should file other bugs). #### investigation (detailed) We are connecting to several content processes during this test (5 in my local tests). The content process that makes this test leak is the content process of the tab targeted by the toolbox. If I skip this specific content process and don't connect to it, there is no leak. The actorIDs of the descriptors are relatively stable across runs, so I simply filtered by processDescriptorFront.actorID to verify this theory. In practice, we already have a DebuggerServer created in this process, because of Tab target of the toolbox. So when we connect to this process target, we create another DebuggerServer in the same process. The content-process.jsm script creates a dedicated loader, which means we really use separate DebuggerServers: https://searchfox.org/mozilla-central/source/devtools/server/startup/content-process.jsm#29-38 If we modify content-process.jsm to use the shared loader instead of creating a new one, the leak disappears as well. Finally, simply setting `invisibleToDebugger` to false is not enough to fix the leak. More interesting I also looked at the test itself to understand which step was leaking precisely. Turns out the test only leaks when you add a breakpoint via `await addBreakpoint(dbg, "asm.js", 7);`. Diving deeper into what leaks in the breakpoint creation, it seems related to a map held by the BreakpointActor: https://searchfox.org/mozilla-central/source/devtools/server/actors/breakpoint.js#40. This map is supposed to be cleared when destroying the actor, but breakpoint actors are never destroyed (at least not in this test). Modifying ThreadActor to destroy all breakpoint actors, fixes the leak for me locally. That being said, I don't understand why this turned into a leak here and not before. Even without enabling windowless service workers, the breakpoint actors are never destroyed. What is it about creating an additional DebuggerServer in the same process that makes this map leak? So far I don't know. I also verified that all the DebuggerServers were correctly destroyed, couldn't spot anything wrong there.