2018938 - Tasks using --gecko-profile extra option resort to outdated symbolication

Status: RESOLVED FIXED

(Testing :: Performance, defect, P2)

Description

[Abhishek Nimalan [:animalan]]

Tasks that use the mozharness extra option --gecko-profile to generate profiles (and do not specify --gecko-profile when running ./mach try) are not provided the dependencies for the newer symbolication (symbolicator-cli) and resort to the fallback (outdated) symbolication method instead. For example, here's a Try push for talos-bcv-profiling.

Comment 1

[Abhishek Nimalan [:animalan]]

Attachment: Bug 2018938 - Provide symbolication dependencies when --gecko-profile is used as a mozharness extra option. r?#perftest

Comment 2

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:sparky, could you have a look please?

For more information, please visit BugBot documentation.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/d72e5b145274
https://hg.mozilla.org/integration/autoland/rev/71d137113c18
Provide symbolication dependencies when --gecko-profile is used as a mozharness extra option. r=perftest-reviewers,taskgraph-reviewers,kshampur,ahal

Comment 4

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/71d137113c18

Comment 5

[Gabriele Svelto [:gsvelto]]

Abhishek we noticed a very large increase in uploads to the symbol server happening around the landing of this patch. Symbol uploads are now more than four times what they used to be. How many tasks do you estimate would have been affected by this change? Is the volume increase expected?

Comment 6

[Abhishek Nimalan [:animalan]]

Thanks for the needinfo!

Context: This patch adds samply and symbolicator-cli (symbolication tools) to tasks that use the --gecko-profile extra option. This allows these tasks to use the CI symbolication introduced in Bug 1970961.

How many tasks do you estimate would have been affected by this change?

~70 tasks as a rough estimate. There are ~22 task definitions with the --gecko-profile extra option and these definitions resolve to about ~70 tasks (so about ~70 tasks are affected). The vast majority of these (~60-65 tasks) are just Talos profiling tasks that need to be triggered manually. The Raptor desktop and mobile profiling tasks (~5 tasks) with --gecko-profile extra option run routinely on mozilla-central.

Is the volume increase expected?

No -- this patch and the Bug 1970961's patch should not be triggering any symbol uploads at all. If they are, this is unintended behaviour and should be fixed. Looking at both patches in question, there doesn't seem to be anything that could trigger symbol uploads (at least to me).

I'll continue monitoring these patches but based off of what I know now, I do not think these patches caused the spike.

Gabriele, if you can provide more detail about the spike (What jobs / tests caused these uploads? What type of symbols are being uploaded? What specific times/dates these spikes occurred? etc.), that would be great -- it might help me confirm whether these patches are the problem.

(Also: I've added to the Matrix discussion on #crashreporting)

Comment 7

[Gabriele Svelto [:gsvelto]]

Clearing the NI? as the elevated number of symbol uploads was not due to this, thanks for the help Abhishek.