Closed Bug 1371014 Opened 3 years ago Closed 3 years ago

talos nightly data is posting to the opt results

Categories

(Testing :: Talos, enhancement)

enhancement
Not set

Tracking

(firefox55 fixed)

RESOLVED FIXED
mozilla55
Tracking Status
firefox55 --- fixed

People

(Reporter: jmaher, Assigned: jmaher)

Details

(Whiteboard: [PI:June])

Attachments

(1 file, 2 obsolete files)

on linux, you can see tp5 has "bi-modal" data, but really it is a data point once/day posting on the opt signature with a much different, but sort of consistent value:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=7776000&series=%5Bmozilla-central,80984697abf1f1ff2b058e2d9f0b351fd9d12ad9,1,1%5D&series=%5Bmozilla-central,f9422672456ec36723cc69e64c10e02cda9dd30f,1,1%5D

this lines up with the 'pgo' series (included on the above link).

In tracing this down, the 'outliers' do match up to the nightly build, and the other data matches up to the opt builds.  This was done by examining the job and matching up the url to download the builds with the url that was referenced from the build job to access the build.

we need to figure out how to get nightly to post to a different series (i.e. not opt, but I am not sure 'pgo' is proper), or consider not running talos on the nightly builds.
:wlach, I recall you had some thoughts on this in the past, any chance they were on this specific topic or you have some new ideas?
Flags: needinfo?(wlachance)
Whiteboard: [PI:June]
1. I believe nightly should be considered a pgo build, it's a treeherder-specific issue if it isn't.
2. Do we really want to run talos against nightlies vs. just m-c pgo? I don't see much value in this.
3. Assuming we do want to run talos against this configuration, we could add an extra property to the series to identify these builds/runs (I guess passed in via mozharness). We already do this for e10s.

https://github.com/mozilla/treeherder/blob/9a5aa7458d03636742f74c6bd94303e150b44e6d/schemas/performance-artifact.json#L81
http://searchfox.org/mozilla-central/rev/20963d7269b1b14d455f47bc0260d0653015bf84/testing/talos/talos/run_tests.py#189
Flags: needinfo?(wlachance)
I would venture to guess that we do not want to run talos on nightly builds, this is easily solved by:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#61
(In reply to Joel Maher ( :jmaher) from comment #3)
> I would venture to guess that we do not want to run talos on nightly builds,
> this is easily solved by:
> https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/test-
> platforms.yml#61

May want to cut out awsy while we're at it.
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #8875431 - Flags: review?(rwood)
thanks :wlach, updated
Attachment #8875431 - Attachment is obsolete: true
Attachment #8875431 - Flags: review?(rwood)
Attachment #8875432 - Flags: review?(rwood)
sorry for the churn, there are a lot of papercuts when digging into this.
Attachment #8875432 - Attachment is obsolete: true
Attachment #8875432 - Flags: review?(rwood)
Attachment #8875442 - Flags: review?(rwood)
Comment on attachment 8875442 [details] [diff] [review]
disable talos + awsy on nightly/devedition

Review of attachment 8875442 [details] [diff] [review]:
-----------------------------------------------------------------

Learn something every day... LGTM!
Attachment #8875442 - Flags: review?(rwood) → review+
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/479b56b160d8
do not run Talos on nightly builds. r=rwood
https://hg.mozilla.org/mozilla-central/rev/479b56b160d8
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
You need to log in before you can comment on or make changes to this bug.