Closed Bug 1509549 Opened 10 months ago Closed 8 months ago

Dynamically load a WASM module that can dump symbol tables for local Firefox for Android builds

Categories

(Core :: Gecko Profiler, enhancement, P2)

enhancement

Tracking

()

RESOLVED FIXED
mozilla67
Tracking Status
firefox67 --- fixed

People

(Reporter: mstange, Assigned: mstange)

References

(Blocks 1 open bug)

Details

Attachments

(4 files, 2 obsolete files)

When profiling local builds, in order to get useful profiles, we need to obtain symbol information from the local binaries.

For example, when you have a local Firefox for Android build, and you push it to a phone and then profile it through WebIDE, we need a way to dump symbols from the libxul.so ELF binary. And for local Firefox for Desktop builds, we also need a way to dump symbols from mach-O binaries (macOS) and PDB files (Windows).
We don't want to ship the code that knows how to read these binaries with Firefox for Desktop because it takes up hundreds of KBs and is only needed by a very small population of users.

In this bug I'm going to prototype a WebAssembly-based solution:
 - have a rust crate that exposes symbol reading APIs
 - compile this crate to wasm
 - host the resulting wasm file on the web somewhere, maybe on resources.perf-html.io, or on res.profiler.firefox.com.
 - land JS code in Firefox which:
   - downloads the wasm file on demand, uses subresource integrity checking to validate its contents
   - maps the binary into wasm memory
   - runs the wasm module in order to get the symbols out of the binary

The JS code in Firefox would also contain the wasm_bindgen-generated JS pieces.
Assignee: nobody → mstange
Status: NEW → ASSIGNED
The WIP patch expects this file to be available at http://127.0.0.1:8080/ . An easy way to make this happen is to "cargo install http-server" and then run http-server in a directory that contains this file.
The patch currently does not work. I'm getting a NetworkError somewhere in here:

> const request = new Request('http://127.0.0.1:8080/profiler_get_symbols_wasm_bg.wasm', {
>   integrity: 'sha384-/Ux6KljNldpNhzaIxcVRzmbhzipgXogi7i7rKZU6U/DEOfShpP2nIoiZv94jS0/d',
>   credentials: 'omit',
> });
> await WebAssembly.compileStreaming(fetch(request));

I'm now going to debug this error.
I found my mistake: I was using "await" where I should have been using "return" or "return await". I made the same mistake in two other places, too. Type annotations would probably have prevented this mistake.
Attachment #9027236 - Attachment is obsolete: true
At the moment, the module can only dump ELF and Mach-O binaries, so it works for
everything but Windows binaries. I'm planning on adding Windows support soon.

The module is currently hosted on https://zealous-rosalind-a98ce8.netlify.com/ ,
which is a netlify server that serves files from the following repo:
https://github.com/mstange/profiler-assets

To make all of this look a bit more official, I'm planning on doing two things:
 - Move the github repo under the devtools-html organization
 - Get a firefox.com subdomain such as profiler-assets.firefox.com for hosting

Depends on D13004
Attachment #9027359 - Attachment is obsolete: true
One advantage of this is that, on Macs that don't have the Xcode command line
tools installed, Xcode will no longer pop up a dialog.

Depends on D13005
Oh, I realize that "ProfilerGetSymbols" is a really clunky name for a module. I'd love to hear suggestions for better names.
No longer depends on: 1509671
Attachment #9027726 - Attachment description: Bug 1509549 - Use ProfilerGetSymbols for mac/linux symbolication instead of calling out to the nm command line tool. r?kmag → Bug 1509549 - Use ProfilerGetSymbols for geckoProfiler WebExtension symbolication and remove all other sources of symbolication. r?kmag
Here's ProfilerGetSymbols-worker.js at work: https://perfht.ml/2DWzAe0
The network cache seems to be caching the wasm module just as I was hoping; here's a profile I took of taking a profile of a local build, while the machine was completely offline: https://perfht.ml/2DXAw1J
The wasm network request shows up in the network monitor as having taken 0.028ms.
Priority: -- → P2

Kris, could you take another look? I've added a "THIS FILE IS AUTOGENERATED" comment at the top of profiler_get_symbols.js. I've also added the following comment in ProfilerGetSymbols.jsm above the URL and the hash:

// Don't let the strange looking URLs and strings below scare you.
// The hash check ensures that the contents of the wasm module are what we
// expect them to be.
// The source code is at https://github.com/mstange/profiler-get-symbols/ .
Flags: needinfo?(kmaglione+bmo)

(In reply to Markus Stange [:mstange] from comment #12)

Kris, could you take another look? I've added a "THIS FILE IS AUTOGENERATED" comment at the top of profiler_get_symbols.js. I've also added the following comment in ProfilerGetSymbols.jsm above the URL and the hash:

Sorry, I didn't realize this was still blocked on me. I'll look at it after this meeting.

Flags: needinfo?(kmaglione+bmo)
Pushed by mstange@themasta.com:
https://hg.mozilla.org/integration/autoland/rev/016e46630cea
Add "WebAssembly" to the list of globals that are present in JSMs. r=Standard8
Pushed by mstange@themasta.com:
https://hg.mozilla.org/integration/autoland/rev/a291f111e24e
Add a ProfilerGetSymbols module which can dump symbols with the help of a dynamically-loaded WebAssembly module. r=kmag
https://hg.mozilla.org/integration/autoland/rev/ba7bb5f95f43
Use ProfilerGetSymbols for geckoProfiler WebExtension symbolication and remove all other sources of symbolication. r=kmag
Status: ASSIGNED → RESOLVED
Closed: 8 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
You need to log in before you can comment on or make changes to this bug.