1509549 - Dynamically load a WASM module that can dump symbol tables for local Firefox for Android builds

Status: RESOLVED FIXED

(Core :: Gecko Profiler, enhancement, P2)

Description

[Markus Stange [:mstange]]

When profiling local builds, in order to get useful profiles, we need to obtain symbol information from the local binaries.

For example, when you have a local Firefox for Android build, and you push it to a phone and then profile it through WebIDE, we need a way to dump symbols from the libxul.so ELF binary. And for local Firefox for Desktop builds, we also need a way to dump symbols from mach-O binaries (macOS) and PDB files (Windows).
We don't want to ship the code that knows how to read these binaries with Firefox for Desktop because it takes up hundreds of KBs and is only needed by a very small population of users.

In this bug I'm going to prototype a WebAssembly-based solution:
 - have a rust crate that exposes symbol reading APIs
 - compile this crate to wasm
 - host the resulting wasm file on the web somewhere, maybe on resources.perf-html.io, or on res.profiler.firefox.com.
 - land JS code in Firefox which:
   - downloads the wasm file on demand, uses subresource integrity checking to validate its contents
   - maps the binary into wasm memory
   - runs the wasm module in order to get the symbols out of the binary

The JS code in Firefox would also contain the wasm_bindgen-generated JS pieces.

Comment 1

[Markus Stange [:mstange]]

Attachment: Bug 1509549 - WIP: Dynamically load WASM code in order to dump symbols.

Comment 2

[Markus Stange [:mstange]]

Attachment: example profiler_get_symbols_wasm_bg.wasm

The WIP patch expects this file to be available at http://127.0.0.1:8080/ . An easy way to make this happen is to "cargo install http-server" and then run http-server in a directory that contains this file.

Comment 3

[Markus Stange [:mstange]]

The patch currently does not work. I'm getting a NetworkError somewhere in here:

> const request = new Request('http://127.0.0.1:8080/profiler_get_symbols_wasm_bg.wasm', {
>   integrity: 'sha384-/Ux6KljNldpNhzaIxcVRzmbhzipgXogi7i7rKZU6U/DEOfShpP2nIoiZv94jS0/d',
>   credentials: 'omit',
> });
> await WebAssembly.compileStreaming(fetch(request));

I'm now going to debug this error.

Comment 4

[Markus Stange [:mstange]]

I found my mistake: I was using "await" where I should have been using "return" or "return await". I made the same mistake in two other places, too. Type annotations would probably have prevented this mistake.

Comment 5

[Markus Stange [:mstange]]

Attachment: Bug 1509549 - WIP: Dynamically load WASM code in order to dump symbols.

Comment 6

[Markus Stange [:mstange]]

Attachment: Bug 1509549 - Add "WebAssembly" to the list of globals that are present in JSMs. r?standard8

Depends on D12836

Comment 7

[Markus Stange [:mstange]]

Attachment: Bug 1509549 - Add a ProfilerGetSymbols module which can dump symbols with the help of a dynamically-loaded WebAssembly module. r?kmag

At the moment, the module can only dump ELF and Mach-O binaries, so it works for
everything but Windows binaries. I'm planning on adding Windows support soon.

The module is currently hosted on https://zealous-rosalind-a98ce8.netlify.com/ ,
which is a netlify server that serves files from the following repo:
https://github.com/mstange/profiler-assets

To make all of this look a bit more official, I'm planning on doing two things:
 - Move the github repo under the devtools-html organization
 - Get a firefox.com subdomain such as profiler-assets.firefox.com for hosting

Depends on D13004

Comment 8

[Markus Stange [:mstange]]

Attachment: Bug 1509549 - Use ProfilerGetSymbols for geckoProfiler WebExtension symbolication and remove all other sources of symbolication. r?kmag

One advantage of this is that, on Macs that don't have the Xcode command line
tools installed, Xcode will no longer pop up a dialog.

Depends on D13005

Comment 9

[Markus Stange [:mstange]]

Oh, I realize that "ProfilerGetSymbols" is a really clunky name for a module. I'd love to hear suggestions for better names.

Comment 10

[Markus Stange [:mstange]]

Here's ProfilerGetSymbols-worker.js at work: https://perfht.ml/2DWzAe0

Comment 11

[Markus Stange [:mstange]]

The network cache seems to be caching the wasm module just as I was hoping; here's a profile I took of taking a profile of a local build, while the machine was completely offline: https://perfht.ml/2DXAw1J
The wasm network request shows up in the network monitor as having taken 0.028ms.

Comment 12

[Markus Stange [:mstange]]

Kris, could you take another look? I've added a "THIS FILE IS AUTOGENERATED" comment at the top of profiler_get_symbols.js. I've also added the following comment in ProfilerGetSymbols.jsm above the URL and the hash:

// Don't let the strange looking URLs and strings below scare you.
// The hash check ensures that the contents of the wasm module are what we
// expect them to be.
// The source code is at https://github.com/mstange/profiler-get-symbols/ .

Comment 13

[Kris Maglione [:kmag]]

(In reply to Markus Stange [:mstange] from comment #12)

Kris, could you take another look? I've added a "THIS FILE IS AUTOGENERATED" comment at the top of profiler_get_symbols.js. I've also added the following comment in ProfilerGetSymbols.jsm above the URL and the hash:

Sorry, I didn't realize this was still blocked on me. I'll look at it after this meeting.

Comment 14

[Markus Stange [:mstange]]

Thanks!

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/016e46630cea
Add "WebAssembly" to the list of globals that are present in JSMs. r=Standard8

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a291f111e24e
Add a ProfilerGetSymbols module which can dump symbols with the help of a dynamically-loaded WebAssembly module. r=kmag
https://hg.mozilla.org/integration/autoland/rev/ba7bb5f95f43
Use ProfilerGetSymbols for geckoProfiler WebExtension symbolication and remove all other sources of symbolication. r=kmag

Comment 17

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/016e46630cea

Comment 18

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/a291f111e24e
https://hg.mozilla.org/mozilla-central/rev/ba7bb5f95f43