on linux, you can see tp5 has "bi-modal" data, but really it is a data point once/day posting on the opt signature with a much different, but sort of consistent value: https://treeherder.mozilla.org/perf.html#/graphs?timerange=7776000&series=%5Bmozilla-central,80984697abf1f1ff2b058e2d9f0b351fd9d12ad9,1,1%5D&series=%5Bmozilla-central,f9422672456ec36723cc69e64c10e02cda9dd30f,1,1%5D this lines up with the 'pgo' series (included on the above link). In tracing this down, the 'outliers' do match up to the nightly build, and the other data matches up to the opt builds. This was done by examining the job and matching up the url to download the builds with the url that was referenced from the build job to access the build. we need to figure out how to get nightly to post to a different series (i.e. not opt, but I am not sure 'pgo' is proper), or consider not running talos on the nightly builds.
:wlach, I recall you had some thoughts on this in the past, any chance they were on this specific topic or you have some new ideas?
1. I believe nightly should be considered a pgo build, it's a treeherder-specific issue if it isn't. 2. Do we really want to run talos against nightlies vs. just m-c pgo? I don't see much value in this. 3. Assuming we do want to run talos against this configuration, we could add an extra property to the series to identify these builds/runs (I guess passed in via mozharness). We already do this for e10s. https://github.com/mozilla/treeherder/blob/9a5aa7458d03636742f74c6bd94303e150b44e6d/schemas/performance-artifact.json#L81 http://searchfox.org/mozilla-central/rev/20963d7269b1b14d455f47bc0260d0653015bf84/testing/talos/talos/run_tests.py#189
I would venture to guess that we do not want to run talos on nightly builds, this is easily solved by: https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#61
(In reply to Joel Maher ( :jmaher) from comment #3) > I would venture to guess that we do not want to run talos on nightly builds, > this is easily solved by: > https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/test- > platforms.yml#61 May want to cut out awsy while we're at it.
Created attachment 8875431 [details] [diff] [review] do not run talos on linux64 nightly builds
Created attachment 8875432 [details] [diff] [review] do not run talos or awsy on linux64-nightly builds thanks :wlach, updated
Created attachment 8875442 [details] [diff] [review] disable talos + awsy on nightly/devedition sorry for the churn, there are a lot of papercuts when digging into this.
Comment on attachment 8875442 [details] [diff] [review] disable talos + awsy on nightly/devedition Review of attachment 8875442 [details] [diff] [review]: ----------------------------------------------------------------- Learn something every day... LGTM!
Pushed by firstname.lastname@example.org: https://hg.mozilla.org/integration/mozilla-inbound/rev/479b56b160d8 do not run Talos on nightly builds. r=rwood