1970961 - CI profile symbolication does not support inlines or the source view - we should use symbolicator-cli for symbolication of Gecko profiles

Status: RESOLVED FIXED

(Testing :: Mozbase, defect, P2)

Description

[Markus Stange [:mstange]]

This bug is about what to do in CI; see bug 1720356 for what should happen when running tests locally.

Edit 2025-07-25: This bug is now "Use symblicator-cli (being added in bug 1969490) for symbolicating Gecko profiles, too, not just for simpleperf profiles".

Original discussion below.

---

We have a ProfilerSymbolicator in testing/mozbase/mozgeckoprofiler/ which we use to symbolicate profiles captured with the Gecko profiler, for example the profiles captured during browsertime pageload tests.

This symbolication code is quite old and hasn't kept pace with recent improvements:

• No inline callstacks
• No source view
• No assembly view

Example: https://share.firefox.dev/4kBmPay

The improvements listed above are available when symbolication is performed on profiler.firefox.com by some JS code in symbolication.js. The implementation is quite a bit more complicated than what we have today in the Python code mentioned earlier.

Here's what happens when you capture a profile with the Firefox UI the normal way:

1. Firefox produces an unsymbolicated JSON file, in the "gecko profile" format.
2. The JSON file is loaded into profiler.firefox.com.
3. profiler.firefox.com converts the JSON into the "processed profile" format.
4. profiler.firefox.com symbolicates the converted profile, with the help of the symbol server.

To fix this bug, we need to find a way for our in-CI symbolication to share code with profiler.firefox.com. It's not practical to create a second implementation in Python of both profile format conversion and symbolication.

In the profiler repo, there is a symbolicator-cli script which is well-suited for our purposes. It needs to be run with node, and it requires a URL to a symbol server, such as the one that samply runs locally when you do samply load profile.json. If we can find a way to integrate this script into testing CI, it would solve this bug.

Comment 1

[Markus Stange [:mstange]]

Hi Greg, how tricky would it be to run a node script from the ProfilerSymbolicator? How would the script be integrated? It's currently the output of a yarn build step that needs to be run in the profiler repo. Would we need to set up something in the profiler repo to publish the "compiled" version of this script to npm?
Do we have other places in the code that use node in a similar fashion?

Comment 2

[Markus Stange [:mstange]]

What we decide to do here will also affect how we handle format changes for the Gecko profile format.

At the moment, changing the Gecko profile format from version N to version N + 1 works like this:

1. We write (but do not land) a patch for Firefox to output the new format. So the patch makes Firefox output different JSON contents (whatever the change actually ends up being), and also makes it put N + 1 into the profile.meta.version field.
2. We write (but do not land) a patch for the front-end in the profiler repo to accept the new format, and an "upgrader" which converts profiles from version N to version N + 1.
3. We get the front-end change reviewed, landed, and deployed.
4. Once the front-end is able to handle version N + 1, we get the Firefox patch reviewed and landed.

If we end up using the symbolicator-cli script, then we'll need to adjust this workflow as follows:

• When deploying the front-end change, we also need to publish a new "compiled" version of symbolicator-cli.
• In the Firefox patch, we need to increment the version of symbolicator-cli we depend on. Or maybe we won't end up using npm and instead just depend on a revision of the profiler repo; in that case we'd need to update a file in the Firefox repo to point to the new profiler revision.

Comment 3

[Greg Mierzwinski [:sparky]]

(In reply to Markus Stange [:mstange] from comment #1)

Hi Greg, how tricky would it be to run a node script from the ProfilerSymbolicator?

Hi :mstange, it wouldn't be too difficult, nearly all of our harnesses except for Talos/AWSY depend on node already.

It's currently the output of a yarn build step that needs to be run in the profiler repo. Would we need to set up something in the profiler repo to publish the "compiled" version of this script to npm?
Do we have other places in the code that use node in a similar fashion?

The browsertime setup might be the closest thing we have to this: https://searchfox.org/mozilla-central/source/tools/browsertime/mach_commands.py

We run that locally for setting up browsertime, and we run it through this shell script in CI to build the package that CI tasks will use: https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/browsertime.sh

I think having a pinned version of the node script/modules will be ideal so it's consistent across mozilla-central pushes. We do that using this package.json file: https://searchfox.org/mozilla-central/source/tools/browsertime/package.json#9

How would the script be integrated?

The code you're referring to is owned by the Profiler team, I recommend discussing that with :canova. Ideally, we shouldn't have to change anything in the test harnesses apart from adding the new node script/modules as dependencies and having it install locally in a manner similar to browsertime or through mach bootstrap.

Comment 4

[Markus Stange [:mstange]]

Thanks for the pointers!

One big difference between browsertime and the symbolicator-cli script seems to be that browsertime doesn't need a build step. The JS files in the browsertime repo are ready to run with node. But symbolicator-cli cannot just be run with node.

Where should the build step be performed, and where should the output ("compiled symbolicator-cli") be stored?

It would not be desirable to store the compiled symbolicator-cli in the profiler repo.

We could store the output on npm, either by manually doing a new release of symbolicator-cli every time we do a deploy, or by adding a github CI job which does a release whenever we merge into the deploy branch.
Or we could have a toolchain job in Firefox CI which pulls the profiler repo, does the build, and outputs the compiled symbolicator-cli as an artifact.

Comment 5

[Markus Stange [:mstange]]

./mach browsertime seems use setup_helper from tools/lint/eslint to perform the actual npm install of the package.json dependencies .

So maybe that's all we need?

• Have a new directory to host the package.json with the dependency on the built symbolicator-cli script, for example at testing/mozbase/mozgeckoprofiler/mozgeckoprofiler/symbolicator-cli/package.json
• In mozbase ProfilerSymbolicator, call setup_helper's package_setup function if needed
• In mozbase ProfilerSymbolicator, invoke the installed symbolicator-cli script

Does that sound reasonable?

The "if needed" above does some heavy lifting - how should we detect whether we need to re-run npm install? Ah, setup_helper seems to use npm ci if no force update is requested, but I think it'll still emit a bunch of logging every time.

Comment 6

[Markus Stange [:mstange]]

Also, here I'm assuming that we've already solved the problem of having samply available, but I guess that's a separate challenge still. I think we'd need to download samply with ./mach bootstrap.

Comment 7

[Kash Shampur [:kshampur] ⌚EST]

Or we could have a toolchain job in Firefox CI which pulls the profiler repo, does the build, and outputs the compiled symbolicator-cli as an artifact.

this might be easiest (and was the first approach that came to my mind) but I am interested in a case against doing this approach

When deploying the front-end change, we also need to publish a new "compiled" version of symbolicator-cli.

Unless i've misunderstood it sounds like we'd always want the latest revision available at time the symbolicator-cli toolchain is (re)built. So this would be opposite of the browsertime approach where we pin a revision.

I might be oversimplifying in my head, but with this approach we probably don't even need to touch or moidfy testing/mozbase/mozgeckoprofiler/mozgeckoprofiler/ at all as we can just directly invoke the new symbolicator-cli (fetched from the hypothetical toolchain) with a python subprocess command

Also, here I'm assuming that we've already solved the problem of having samply available, but I guess that's a separate challenge still. I think we'd need to download samply with ./mach bootstrap.

We have a samply toolchain now (bug 1967836) just in case you were not aware yet (unless you meant something else by this statement). What/why would we need mach bootstrap?

Comment 8

[Kash Shampur [:kshampur] ⌚EST]

but pinning version makes most sense... easier to debug in future

Comment 9

[Kash Shampur [:kshampur] ⌚EST]

I am a bit unclear on the N, N+1 thing though

and after talking to sparky, i misspoke, we should update mozgeckoprofiler as many things would be using it (i mistakenly assumed prioritizing simpleperf tasks)

Comment 10

[Greg Mierzwinski [:sparky]]

(In reply to Markus Stange [:mstange] from comment #4)

We could store the output on npm, either by manually doing a new release of symbolicator-cli every time we do a deploy, or by adding a github CI job which does a release whenever we merge into the deploy branch.
Or we could have a toolchain job in Firefox CI which pulls the profiler repo, does the build, and outputs the compiled symbolicator-cli as an artifact.

Having a pre-built release that we could pull would make its usage mostly trivial, and it could become a fetch task instead of a toolchain. Although a toolchain might be preferred if we ever need anything special for Mozilla-related tasks.

Where should the build step be performed, and where should the output ("compiled symbolicator-cli") be stored?

We can do the same thing as browsertime where we install directly to the same folder that the mach_commands.py (or install scripts exist in) and add them to the .gitignore/.hgignore files. Alternatively, you can always install to a ~/.mozbuild subfolder.

(In reply to Markus Stange [:mstange] from comment #5)

./mach browsertime seems use setup_helper from tools/lint/eslint to perform the actual npm install of the package.json dependencies .

So maybe that's all we need?

• Have a new directory to host the package.json with the dependency on the built symbolicator-cli script, for example at testing/mozbase/mozgeckoprofiler/mozgeckoprofiler/symbolicator-cli/package.json
• In mozbase ProfilerSymbolicator, call setup_helper's package_setup function if needed
• In mozbase ProfilerSymbolicator, invoke the installed symbolicator-cli script

Does that sound reasonable?

Yup, that sounds reasonable to me. It's unclear to me though why we can't call setup_helper.package_setup in a toolchain and then pull in the artifact of that setup to our CI tasks? That would save a lot of CI time even if it's platform specific. Maybe I'm misunderstanding what you're considering here. I think for local runs, it makes sense to have something in (or near) mozgeckoprofiler to handle the installs.

The "if needed" above does some heavy lifting - how should we detect whether we need to re-run npm install? Ah, setup_helper seems to use npm ci if no force update is requested, but I think it'll still emit a bunch of logging every time.

That shouldn't be difficult. We do this for detecting when to update the browsertime package, and it's been working without issue for a while now: https://searchfox.org/mozilla-central/source/testing/raptor/mach_commands.py#194-201

Comment 11

[Markus Stange [:mstange]]

In bug 1969490, Abhishek is doing a lot of the work that's needed here. He's adding a task which outputs the built symbolicator-cli as an artifact.

Comment 12

[Abhishek Nimalan [:animalan]]

Attachment: WIP: Bug 1970961 - Update mozgeckoprofiler CI symbolication. r=kshampur,perftest-reviewers

Comment 13

[Abhishek Nimalan [:animalan]]

Attachment: Bug 1970961 - Add macOS (x86-64 and ARM) and Windows samply toolchains. r=kshampur,#perftest-reviewers

Comment 14

[Abhishek Nimalan [:animalan]]

Attachment: Bug 1970961 - Add symbolication dependencies to profiling tasks using mozgeckoprofiler. r=kshampur,#perftest-reviewers

Comment 15

[Abhishek Nimalan [:animalan]]

Attachment: Bug 1970961 - Update mozgeckoprofiler CI symbolication. r=kshampur,#perftest-reviewers

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/cf3790ae281e
https://hg.mozilla.org/integration/autoland/rev/7639ae467b5f
Add macOS (x86-64 and ARM) and Windows samply toolchains. r=kshampur,perftest-reviewers
https://github.com/mozilla-firefox/firefox/commit/96b0230b7730
https://hg.mozilla.org/integration/autoland/rev/18165e60746c
Add symbolication dependencies to profiling tasks using mozgeckoprofiler. r=kshampur,perftest-reviewers,taskgraph-reviewers,ahal
https://github.com/mozilla-firefox/firefox/commit/6a23566eaf3c
https://hg.mozilla.org/integration/autoland/rev/94b59f1df154
Update mozgeckoprofiler CI symbolication. r=perftest-reviewers,kshampur

Comment 17

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/7639ae467b5f
https://hg.mozilla.org/mozilla-central/rev/18165e60746c
https://hg.mozilla.org/mozilla-central/rev/94b59f1df154