1605366 - consider running condprof jobs once/day instead of every m-c push

Status: NEW

(Testing :: Performance, task, P3)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we have these cool condprof jobs that generate profiles that are used for raptor runs to make a more realistic perf result.

currently these run for each platform (linux, osx, win, android arm7, android aarch64) on every mozilla-central push as well as a once/day cronjob (which then duplicates the jobs on all 5 platforms.

The original intention was to run this once/day instead of 8-10 times/day. We should ensure there are no other drivers for running this per m-c commit. It could be there are difficulties running jobs only on a crontab and not per m-c commit.

Assuming the original desire of 1x/day holds true, we should figure out a way to have these condprof jobs run only 1x/day. I think maybe a happy medium would be 2x/day on the nightly scheduler if that would be simpler.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Dave and Tarek do you have any feedback regarding this request?

Comment 2

[Tarek Ziadé (:tarek)]

If there's a way to avoid them to run on mozilla-central pushes, happy to change this. As far as I know, these pushes runs everything.

The only cases where we want to have them running are:

• manual mach try runs (via fuzzy --full)
• the cron jobs

maybe there's a way to catch some environment info to know that we're doing a m-c push, and discard the run?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

can you fix this with run-on-projects: ['try'] ?

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

As long as there's no significantly increased risk from having the age of the profiles used in perf testing increasing to 24 hours then I have no concerns here. It sounds like we would just need to make these run on try and in the nightly cron.

Comment 5

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #3)

can you fix this with run-on-projects: ['try'] ?

IIRC a job can be ran on try with the option --full without run-on-projects: ['try']. I am saying this because tarek mentioned this when referring to try.

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #4)

As long as there's no significantly increased risk from having the age of the profiles used in perf testing increasing to 24 hours then I have no concerns here. It sounds like we would just need to make these run on try and in the nightly cron.

I see no nightly for android in the cron tasks, only for desktop.
The condprof jobs are:

"condprof-android-hw-g5-7-0-arm7-api-16-fenix",
"condprof-android-hw-g5-7-0-arm7-api-16-fennec68",
"condprof-android-hw-g5-7-0-arm7-api-16-geckoview",
"condprof-android-hw-p2-8-0-aarch64-fenix",
"condprof-android-hw-p2-8-0-aarch64-fennec68",
"condprof-android-hw-p2-8-0-android-aarch64-geckoview",
"condprof-linux64-firefox",
"condprof-macosx64-firefox",
"condprof-windows2012-64-firefox",

I have the feeling that I am missing something.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Please note that with running it only once a day we may hit the update interval for various services like safebrowsing. That could have negative impact on the performance metrics as visible in perfherder.

I think we should make sure to have bug 1626604 (especially comment 13 and following solved first).

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Alexandru Ionescu (needinfo me) :alexandrui from comment #5)

I have the feeling that I am missing something.

Here's an example where the m-c and cron condprof jobs have occured against the same push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=condprof&revision=1b10b4ffe94deab9dc1f93e093c64b123cb25072.

As :whimboo suggests, let's wait for bug 1636956 until we work on this. In the meantime, :jmaher could you provide :alexandrui with some direction for where these tasks are scheduled via cron, and where we might make the change to put these on the nightly scheduler?

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

here is the cron.yml file:
https://searchfox.org/mozilla-central/source/.cron.yml#230

plenty of examples in there to figure out changes.

Comment 9

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Thank you, Joel. I got this :) there were some extra places I had to look over.