1905952 - Use MOZ_PROFILER_SYMBOLICATE or similar to symbolicate profiles when we use signal-controlled profiling in CI.

Status: NEW

(Core :: Gecko Profiler, task, P3)

Description

[Adam Brouwers-Harries [:aabh] [he/him] ⌚GMT]

Bug 1883903 will give us the ability to capture profiles when Firefox hangs during a mochitest in CI. These profiles, however, are not symbolicated - making them less useful for debugging purposes. Manually symbolicating them would be an involved and labourious process, so we should (instead) symbolicate them before we upload them as an artifact.

Using MOZ_PROFILER_SYMBOLICATE is the obvious answer to this. Either setting it before every mochitest, or manually setting it when we start the profiler with a signal, and hoping it propagates across processes.

Comment 1

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

Currently you can symbolicate the ones you captured by uploading the symbols to the symbol server via treeherder and then clicking on the "Re-symbolicate profile" button inside the profile info panel in the profiler.

I haven't used MOZ_PROFILER_SYMBOLICATE at all as far as I remember, but the comment here says that it's very expensive during the profile capturing process. Uploading symbols can happen retroactively, so you don't have to symbolicate all the profiles while capturing that way. To be able to upload the symbols:

• Go to the treeherder push you want to symbolicate and make sure you're logged in.
• Click the action menu (little triangle) on the top right side.
• Click on action "Add new jobs" or "Add new jobs (search)". (I will be explain "Add new jobs (search)" in the following steps)
• Search for '-upload-symbols (including the apostrophe to make non-fuzzy match)
• Then choose the task depending on your profile's platform+build type. For example, linux opt build will be build-linux64/opt-upload-symbols.
• Then add the selected task and click on trigger.

This will trigger the upload task and once it's done and we will be able to re-symbolicate. But it will cause a pause for the developers if they want to analyze an issue. And looking at the steps to do this, it's probably something all of our engineers know (Maybe we can document it somewhere?).

But if it turns out that MOZ_PROFILER_SYMBOLICATE is something cheaper than I thought, maybe it could be fine to have it?

Comment 2

[Florian Quèze [:florian]]

Is MOZ_PROFILER_SYMBOLICATE producing readable profiles? I think last time I tried it for shutdown profiles, the results were quite disappointing.

I wonder if making it easier to upload symbols for a try build wouldn't be a better path forward.

Comment 3

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

(In reply to Florian Quèze [:florian] from comment #2)

Is MOZ_PROFILER_SYMBOLICATE producing readable profiles? I think last time I tried it for shutdown profiles, the results were quite disappointing.

That's a good point too. I don't remember looking at a profile that was symbolicated by this before.

I wonder if making it easier to upload symbols for a try build wouldn't be a better path forward.

I would prefer this one as well.

Comment 4

[Adam Brouwers-Harries [:aabh] [he/him] ⌚GMT]

Now that we're seeing profiles in CI, I wonder if using MOZ_PROFILER_SYMBOLICATE would be useful in the short term, until we can upload symbols properly? At least, it would strictly increase the information that developers have - it wouldn't make anything worse.

Comment 5

[Adam Brouwers-Harries [:aabh] [he/him] ⌚GMT]

Attachment: WIP: Bug 1905952 - Turn on MOZ_PROFILER_SYMBOLICATE for mochitests, when we're not explicitly running the profiler