1786400 - Run raptor browsertime jobs with an additional gecko profiler enabled run

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P2)

Description

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

We wanted to enable the profiler for all raptor browsertime tests by default in a low overhead in treeherder. But after some investigations, it appeared that i's not straightforward to enable the profiler all the time because some test overheads were vastly different than others. Like some tests were 10-20% increased in some and more than 200% increased in some other tests and they are different per platform as well.
So we've decided to add another profiler enabled run after this, so we will still be able to capture profiles easily.

Comment 1

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

Attachment: Bug 1786400 - Add the renderer thread to default threads list for raptor r?sparky,#perftest

Comment 2

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

Attachment: Bug 1786400 - Add --extra-profiler-run command line argument for raptor r?sparky,#perftest

Depends on D155278

Comment 3

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

Attachment: Bug 1786400 - Add an extra browsertime run if extra_profiler_run is true r?sparky,#perftest

Depends on D155279

Comment 4

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

Attachment: Bug 1786400 - Collect extra_profiler_run profiles properly r?sparky,#perftest

Depends on D155280

Comment 5

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

Attachment: Bug 1786400 - Do not raise an exception if symbolication fails r?sparky,#perftest

Depends on D155281

Comment 6

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

Attachment: Bug 1786400 - Do not raise an exception if profiles are not found for extra_profiler_run r?sparky,#perftest

Depends on D155282

Comment 7

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

Attachment: Bug 1786400 - Enable the extra-profiler-run in taskcluster r?sparky,#perftest

Depends on D155283

Comment 8

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

Hi Joel,
My patches that will enable the extra profiler run for raptor browsertime tests are nearly ready to land after the last set of reviews. Here's my latest try run: https://treeherder.mozilla.org/jobs?repo=try&revision=327141447d3d2714f1c4d8037fc61dd11c5a9d84
After talking with :sparky, we've decided to not enable this extra run for youtube playback tests because it was doubling the running times (from ~24mins to ~50mins) which wasn't great.
Considering this, I believe this will be a lot less overhead for the osx build pool. I intend to enable this for osx builds as well as a first step but wanted to get your feedback as well. Do you think it's worth/feasable to start with enabling this for osx as well? I intend to monitor the overhead we will bring to our build pool and we can reduce the number of iterations if we need to in the osx tasks

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

isn't the average increase per job 1 minute? are you saying without youtube the average increase is much less? Right now our OSX pool is backlogged 8+ hours every day, so I am going to be very cautious about adding more load to OSX.

It is ok to move forward, but please monitor this and look at the usage:
https://earthangel-b40313e5.influxcloud.net/d/avHECHgMk/dc-usage-workertypes?orgId=1&var-provisioner=releng-hardware&var-workerType=gecko-t-osx-1015-r8&from=now-7d&to=now

please note that OSX usage is both unittests and perftests on the same pool (different from windows/linux/android)

Comment 10

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

Oh sorry I learned that these tests were a bit special because I learned from other perf testing folks that we don't run these tests by default for sheriffing, they are run a lot less frequently and I wasn't running them while testing initially. So we can just ignore those tests for now.
Current average increase is 1 minute per job with our default raptor browsertime tests indeed.

I will be monitoring the usage there and will take action in case we it becomes an issue.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4ac818280b21
Add the renderer thread to default threads list for raptor r=sparky,perftest-reviewers
https://hg.mozilla.org/integration/autoland/rev/fe956777eaa4
Add --extra-profiler-run command line argument for raptor r=sparky,perftest-reviewers
https://hg.mozilla.org/integration/autoland/rev/a17f6e8afdad
Add an extra browsertime run if extra_profiler_run is true r=sparky,perftest-reviewers
https://hg.mozilla.org/integration/autoland/rev/bbef7c1a87ee
Collect extra_profiler_run profiles properly r=sparky,perftest-reviewers
https://hg.mozilla.org/integration/autoland/rev/952d2ac08415
Do not raise an exception if symbolication fails r=sparky,perftest-reviewers
https://hg.mozilla.org/integration/autoland/rev/558ab82b7681
Do not raise an exception if profiles are not found for extra_profiler_run r=sparky,perftest-reviewers
https://hg.mozilla.org/integration/autoland/rev/5c2c844dbdc1
Enable the extra-profiler-run in taskcluster r=sparky,perftest-reviewers

Comment 12

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/4ac818280b21
https://hg.mozilla.org/mozilla-central/rev/fe956777eaa4
https://hg.mozilla.org/mozilla-central/rev/a17f6e8afdad
https://hg.mozilla.org/mozilla-central/rev/bbef7c1a87ee
https://hg.mozilla.org/mozilla-central/rev/952d2ac08415
https://hg.mozilla.org/mozilla-central/rev/558ab82b7681
https://hg.mozilla.org/mozilla-central/rev/5c2c844dbdc1