1425850 - collect perf profiles on a regular basis so we have baselines to look at

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

currently we would need to push to try or run locally, if we could run talos on nightly builds and collect perf.io profiles that are uploaded as artifacts, then we could investigate changes that sneak in over time easier.

possibly we do this on the codecov builds on mozilla-central since we already do not care about the timing results there :)

Comment 1

[Robert Wood [:rwood]]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #0)
> currently we would need to push to try or run locally, if we could run talos
> on nightly builds and collect perf.io profiles that are uploaded as
> artifacts, then we could investigate changes that sneak in over time easier.
> 
> possibly we do this on the codecov builds on mozilla-central since we
> already do not care about the timing results there :)

I wonder how much impact ccov instrumentation has on profiling though i.e. how different a profile would look. I'm wondering if we should do this on regular opt builds say on inbound or central, but then don't output any PERFHERDER_DATA json object at the end so that the results don't actually get ingested, but the profiles would be archived. We might need to add a new flag to tell talos not to output perfherder data... or just do that by default if running with the --geckoProfile flag?

Perhaps we could also use SETA to trigger this every x push or something. Just ideas...

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

good point- we could schedule on the nightly scheduler (keep it simple) a new test type that is T-P(...) which has all the profiles.  Of course we need to make them discoverable :)

Comment 3

[Robert Wood [:rwood]]

Oh that's cool I never even know there were tests that just run on a nightly basis, I found them though hah:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=nightly

And I'm assuming these are the configs (for linux):

https://searchfox.org/mozilla-central/rev/88439dc6c5b02e167032374071cdc697a056efa1/taskcluster/ci/test/test-platforms.yml#57

So if we enable talos there (in a new T-P test group) does it even matter if we submit it to perfherder then since it will be marked as a different platform anyway?

Or should I also make a patch in talos so that if running on production (but not try) then don't dump out PERFHERDER_JSON? Maybe it's better not to submit if we won't need the data anyway.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

we shouldn't load data in perfdata, I believe we already skip perfherder_data output if geckoProfile is going. You have the right stuff for nightly- although I would ensure it is limited to mozilla-central to be safe.

Comment 5

[Robert Wood [:rwood]]

Ok sounds like a good plan, thanks - and yes you are correct, re: there's already no perherder data output when profiling, awesome:

https://searchfox.org/mozilla-central/rev/88439dc6c5b02e167032374071cdc697a056efa1/testing/talos/talos/output.py#206

Comment 6

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bda38fcd17ee4e3a640a3f45f824916d8a6def5e

Comment 7

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=dfcb31c02ffae9fa0da9224cb57f4f1e7ea9564f

Comment 8

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=26591d42dc271bc9782f50a85c07b41eb9dff8fb

Comment 9

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ab135c73148a32a9a9dba4809d1fd7fe02c4f072

Comment 10

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2486a187cd9f24c7a5a596f70ef698ad1bf59426

Comment 11

[Robert Wood [:rwood]]

Attachment: Bug 1425850 - Add new talos gecko profiling job on osx nightly;

Review commit: https://reviewboard.mozilla.org/r/214018/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/214018/

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8943642 [details]
Bug 1425850 - Add new talos gecko profiling job on osx nightly;

https://reviewboard.mozilla.org/r/214018/#review219722

please ensure all tests are only run on central/try, beta seems ok as well, but possibly not necessary.

One thought- since these are duplicated tests with just one extra option- could we figure out a way to make a transform that adds the one option and adjusts the branches?

::: taskcluster/ci/test/talos.yml:87
(Diff revision 1)
> +    try-name: dromaeojs-profiling
> +    treeherder-symbol: T-P(d)
> +    run-on-projects:
> +        by-test-platform:
> +            .*-qr/.*: ['mozilla-central', 'try']
> +            default: ['mozilla-beta', 'mozilla-central', 'mozilla-inbound', 'autoland', 'try']

this is on inbound/autoland, that doesn't seem right

::: taskcluster/ci/test/talos.yml:128
(Diff revision 1)
> +    try-name: g1-profiling
> +    treeherder-symbol: T-P(g1)
> +    run-on-projects:
> +        by-test-platform:
> +            .*-qr/.*: ['mozilla-central', 'try']
> +            default: ['mozilla-beta', 'mozilla-central', 'mozilla-inbound', 'autoland', 'try']

same here, this is on all branches.

Comment 13

[Robert Wood [:rwood]]

Comment on attachment 8943642 [details]
Bug 1425850 - Add new talos gecko profiling job on osx nightly;

https://reviewboard.mozilla.org/r/214018/#review219722

Good call re: run on central/try thanks I missed that.

re: transform, IMO I think it's best to keep it the same as our other test definitions just for simplicity and flexibility, i.e. this way it lets us increase the max run time on an individual basis in case one of the tests takes longer with profiling on, etc.

Comment 14

[Robert Wood [:rwood]]

After discussing on IRC, we'll go with this now and in the future we may revisit and try using transforms for our talos test variations like this one and the stylo flavours etc.

Comment 15

[Robert Wood [:rwood]]

Comment on attachment 8943642 [details]
Bug 1425850 - Add new talos gecko profiling job on osx nightly;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/214018/diff/1-2/

Comment 16

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2a9c482c888764e6e35d6e7a3896cf6a6d96c5a8

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8943642 [details]
Bug 1425850 - Add new talos gecko profiling job on osx nightly;

https://reviewboard.mozilla.org/r/214018/#review219872

looking forward to seeing these profiles make fixing perf regressions easier!

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8943642 [details]
Bug 1425850 - Add new talos gecko profiling job on osx nightly;

https://reviewboard.mozilla.org/r/214018/#review219874

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8943642 [details]
Bug 1425850 - Add new talos gecko profiling job on osx nightly;

https://reviewboard.mozilla.org/r/214018/#review219876

4th time is a charm?

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/19c1313a1dd5
Add new talos gecko profiling job on osx nightly; r=jmaher

Comment 21

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/19c1313a1dd5