1683758 - [macOS 11] The profiler does not obtain symbols for system libraries starting with macOS Big Sur

Status: RESOLVED FIXED

(Core :: Gecko Profiler, defect, P3)

Description

[Markus Stange [:mstange]]

Symbolication of macOS system libraries currently only works if the Mozilla symbol server has the symbols. If that's not the case, for example when running a Beta version of macOS, the local symbol lookup is unsuccessful, because most system libraries no longer exist as files on the disk; they only exist in the dyld shared cache.

Example profile without symbols: https://share.firefox.dev/3mFof55

To fix this, we will either need to add the ability to extract symbols from the dyld shared cache, or we will need to somehow get the symbol table from the mapping of the library that is currently loaded into the current process's memory.

I have filed https://github.com/gimli-rs/object/issues/268 in support of this.

Comment 1

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

To clarify: Is the issue only with beta OSes? If yes, you may want to edit the summary to mention "beta", thanks.
Setting priority to P3, assuming that it doesn't affect users on non-Beta macOS.

But if in fact it does affect all Big Sur versions, feel free to change to P2. (Or also if that blocks us from doing important work there.)

Comment 2

[Markus Stange [:mstange]]

The inability to obtain the symbols locally affects all Big Sur versions. But on non-Beta versions this is mitigated by the fact that we have symbols of macOS system libraries on the Mozilla symbol server, so the local symbol lookup code doesn't even run for system libraries.
I agree with P3.

However, we should be aware that the symbol server import happens on a best-effort basis and may not be as timely, reliable and universal as we want it to be. So it would be nice if we didn't have to rely on it.

Comment 3

[Markus Stange [:mstange]]

Attachment: Bug 1683758 - Update profiler-get-symbols to https://github.com/mstange/profiler-get-symbols/commit/007c254385f74ed0538f6e915c73c5055f194a8c . r=canaltinova

This update adds support for macOS 12 compatible dyld shared cache parsing.
It also improves symbolication performance for large PDB files, unrelated to this bug.

Comment 4

[Markus Stange [:mstange]]

Attachment: Bug 1683758 - For macOS system libraries, fall back to symbolicating from the dyld shared cache. r=canaltinova

This lets us obtain symbols for macOS system libraries on macOS 11+
even if these symbols are not present on the Mozilla symbol server.

Some background for this is described in https://github.com/gimli-rs/object/issues/268 .

This patch makes use of the syntax dyldcache:<dyldcachepath>:<librarypath>.
There is some code in the profiler-get-symbols wasm integration which parses this syntax
and turns it into a CandidatePathInfo::InDyldCache enum value.
And profiler-get-symbols itself will then check the dyld shared cache for the requested
library, and parse it from there.

You can run the following pieces of code in the Firefox error console to
test whether this patch is working, on macOS 11 or above:

var { createLocalSymbolicationService } = ChromeUtils.import("resource://devtools/client/performance-new/symbolication.jsm.js");
var service = createLocalSymbolicationService(Services.profiler.sharedLibraries, []);
var appkit = Services.profiler.sharedLibraries.find(l => l.name == "AppKit");
var [addrs, index, buffer] = await service.getSymbolTable(appkit.debugName, appkit.breakpadId);
addrs.length

var { createLocalSymbolicationService } = ChromeUtils.import("resource://devtools/client/performance-new/symbolication.jsm.js");
var service = createLocalSymbolicationService(Services.profiler.sharedLibraries, []);
var appkit = Services.profiler.sharedLibraries.find(l => l.name == "AppKit");
JSON.parse(await service.querySymbolicationApi("/symbolicate/v5", JSON.stringify({memoryMap:[[appkit.name,appkit.breakpadId]],stacks:[[[0,0x12f00d]]]}))).results[0].stacks[0][0]

Before this patch, getSymbolTable would throw an error (file not found), and
querySymbolicationApi would return an object without a function name.
With this patch, getSymbolTable finds all the symbols in AppKit, and
querySymbolicationApi returns the correct function name.

Depends on D123815

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/15834d220c60
Update profiler-get-symbols to https://github.com/mstange/profiler-get-symbols/commit/007c254385f74ed0538f6e915c73c5055f194a8c . r=canaltinova
https://hg.mozilla.org/integration/autoland/rev/cf7d09be4025
For macOS system libraries, fall back to symbolicating from the dyld shared cache. r=canaltinova

Comment 6

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/15834d220c60
https://hg.mozilla.org/mozilla-central/rev/cf7d09be4025