Open Bug 1939998 Opened 2 months ago Updated 9 days ago

Ship a cycle collector log analyzer in the browser

Categories

(Core :: Cycle Collector, enhancement)

Product:
Core
Component:
Cycle Collector
Type:
enhancement

Tracking

()

People

(Reporter: mayankleoboy1, Unassigned)

References

(
URL
)

Details

Attachments

(1 file)

cc-find-rrot output for comment 7
312.08 KB, text/plain
Details

Asking reporters to generate cc/gc logs (which get multi-gb in size) is tedious, timetaking, and doesnt work most of the time.

Consider rewriting DMD (or whatever tool is used) in something that can be run directly inside the browser.
The user should only click a button. And the tool will generate cc/gc logs and process them right within the browser. Then it can take only the relavent info from the graphs, and create a concise report/dump that can be uploaded to bmo.

I am definitely not the first person to think of it, so please dupe to existing bugs.

This doesn't really have anything to do with DMD or WASM. I guess ideally we would analyze DMD+CC logs but supporting DMD would be trickier. Olli did implement a CC log analyzer in JS but that was a long time ago.

Summary: Consider rewriting DMD in WASM and shipping with the Browser → Ship a cycle collector log analyzer in the browser

I threw together a prototype of this:
https://cc-find-roots.netlify.app/

I converted find-roots.py to Rust using chat.deepseek.com, fixed it up and got it working in wasm. It likely has bugs but seems to sort of work.

Source is here: https://github.com/jrmuizel/wasm-find-roots

What is the best way to capture cc logs?

  • What i saw is that the logs are created and then the actual cc runs.
  • The "save concise logs" or "save verbose logs" buttons should ideally have the functionality of "run cc and save logs"
  • There is no way to associate the multiple cc logs with a specific process/tab
  • What do we do with the gc logs? Are they useful?
  • Maybe the tool can pickup more files from the disk as it needs them? The user only should point the tool at the directory where logs are saved
  • Or maybe the user selects ALL the cc-log files and then the tool can use whatever it needs

I am using the STR from bug 1937398. I get the large memory increase, and then I go to about:memory and click on "save verbose logs". That saved a bunch of log files, but the actual cc was not triggered - the cc got triggered after maybe 15 seconds after the logs had been completely written to disk.

I just updated the site to look for nsGlobalWindowInner instead of JSWindowActorProtocol. You can try again. (and probably attach the output instead of putting it in a comment if it's large)

Same cc-log as before
Profile: https://share.firefox.dev/4gObMZy (significantly reduced time ti run)
Output: Much larger, in the attachment.

When trying to analyze a 3.08GB (incomplete-cc) log file, the app allocates 3-4GB RAM, dies and releases the memory. I get this in the console:

10:12:49.332 injected firefox content.js:142:11
10:13:00.853 Uncaught (in promise) RuntimeError: unreachable executed
passArray8ToWasm0 https://cc-find-roots.netlify.app/pkg/wasm_find_roots.js:87
count_lines https://cc-find-roots.netlify.app/pkg/wasm_find_roots.js:101
run https://cc-find-roots.netlify.app/main.js:14
wasm_find_roots_bg.wasm:990753:1

Yeah, that's expected. I need to figure out how to build a wasm64 version

The memory usage could probably be massively reduced by not storing the edge names. That's a huge chunk of the runtime on the Python version. Ideally the output here would be very simple so a reasonably informed user could understand what information it is revealing. I think to answer the basic question of "why the heck is my Firefox leaking windows?" we really only need the description of the initial and final nodes. (The information about the final node would help us figure out if this is a leaking window or some other random window that happens to be in the CC graph.) So, we'd skip all of the information about the path (including the edge names) and the "known edges" is probably not very useful and can be rather spammy (you can usually recreate that via code audit). The "Root is a ref counted object with 1 unknown edge(s)." part is pretty short so maybe that would be okay to include.

In the specific case of bug 1939295, I think that getting the class name, FetchStreamReader, from Katelyn's logs was sufficient for Kagami to figure out a possible problem here, and would be sufficient to see if these other issues people are experiencing are the same.

All of that other information in the regular find_roots output is useful, but mostly for correlating with other sources of information, like a DMD or refcount log, which we're not going to have access to here.

See Also: → 1939295

We do have a way already to write a CC log analyzer in JS, via the nsICycleCollectorListener.idl interface, but I think only Olli has really ever used it, and probably not for a number of years.

As an example of how long it has been since anybody used nsICycleCollectorListener.idl, I happened across these comments from a decade ago in the code today:

  • "We don't support after-processing for weak map entries."
  • "We don't support after-processing for incremental roots."

So the interface would have to be updated to support those if we wanted to do it that way.

We could also just have log analyzer in the browser, basically the functionality of heapgraph, but keeping the graph around for filtering and such.
https://mozilla.pettay.fi/cclog.html is an early wip/poc, doesn't work with logs created on Windows (because the addresses use different syntax), may not really work with GC logs and has only one way to find roots, heapgraph has, IIRC, couple of algorithms etc. So, WIP/POC :)

If we could get this working, then i think an interesting followup would be to generate telemetry about what is causing ghost windows. We could do something like, if there's a ghost window in a process, run the analyzer and report the name of the CC class as telemetry. I think if we run it with !WantDebugInfo() it won't include any sensitive information, but of course we'd have to verify that. Ghost window telemetry is nice, but it doesn't really give you anything actionable by itself.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: