talos nightly data is posting to the opt results

RESOLVED FIXED in Firefox 55

Status

Testing
Talos
RESOLVED FIXED
5 months ago
5 months ago

People

(Reporter: jmaher, Assigned: jmaher)

Tracking

Trunk
mozilla55
Points:
---

Firefox Tracking Flags

(firefox55 fixed)

Details

(Whiteboard: [PI:June])

Attachments

(1 attachment, 2 obsolete attachments)

(Assignee)

Description

5 months ago
on linux, you can see tp5 has "bi-modal" data, but really it is a data point once/day posting on the opt signature with a much different, but sort of consistent value:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=7776000&series=%5Bmozilla-central,80984697abf1f1ff2b058e2d9f0b351fd9d12ad9,1,1%5D&series=%5Bmozilla-central,f9422672456ec36723cc69e64c10e02cda9dd30f,1,1%5D

this lines up with the 'pgo' series (included on the above link).

In tracing this down, the 'outliers' do match up to the nightly build, and the other data matches up to the opt builds.  This was done by examining the job and matching up the url to download the builds with the url that was referenced from the build job to access the build.

we need to figure out how to get nightly to post to a different series (i.e. not opt, but I am not sure 'pgo' is proper), or consider not running talos on the nightly builds.
(Assignee)

Comment 1

5 months ago
:wlach, I recall you had some thoughts on this in the past, any chance they were on this specific topic or you have some new ideas?
Flags: needinfo?(wlachance)
(Assignee)

Updated

5 months ago
Whiteboard: [PI:June]
1. I believe nightly should be considered a pgo build, it's a treeherder-specific issue if it isn't.
2. Do we really want to run talos against nightlies vs. just m-c pgo? I don't see much value in this.
3. Assuming we do want to run talos against this configuration, we could add an extra property to the series to identify these builds/runs (I guess passed in via mozharness). We already do this for e10s.

https://github.com/mozilla/treeherder/blob/9a5aa7458d03636742f74c6bd94303e150b44e6d/schemas/performance-artifact.json#L81
http://searchfox.org/mozilla-central/rev/20963d7269b1b14d455f47bc0260d0653015bf84/testing/talos/talos/run_tests.py#189
Flags: needinfo?(wlachance)
(Assignee)

Comment 3

5 months ago
I would venture to guess that we do not want to run talos on nightly builds, this is easily solved by:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#61
(In reply to Joel Maher ( :jmaher) from comment #3)
> I would venture to guess that we do not want to run talos on nightly builds,
> this is easily solved by:
> https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/test-
> platforms.yml#61

May want to cut out awsy while we're at it.
(Assignee)

Comment 5

5 months ago
Created attachment 8875431 [details] [diff] [review]
do not run talos on linux64 nightly builds
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #8875431 - Flags: review?(rwood)
(Assignee)

Comment 6

5 months ago
Created attachment 8875432 [details] [diff] [review]
do not run talos or awsy on linux64-nightly builds

thanks :wlach, updated
Attachment #8875431 - Attachment is obsolete: true
Attachment #8875431 - Flags: review?(rwood)
Attachment #8875432 - Flags: review?(rwood)
(Assignee)

Comment 7

5 months ago
Created attachment 8875442 [details] [diff] [review]
disable talos + awsy on nightly/devedition

sorry for the churn, there are a lot of papercuts when digging into this.
Attachment #8875432 - Attachment is obsolete: true
Attachment #8875432 - Flags: review?(rwood)
Attachment #8875442 - Flags: review?(rwood)

Comment 8

5 months ago
Comment on attachment 8875442 [details] [diff] [review]
disable talos + awsy on nightly/devedition

Review of attachment 8875442 [details] [diff] [review]:
-----------------------------------------------------------------

Learn something every day... LGTM!
Attachment #8875442 - Flags: review?(rwood) → review+

Comment 9

5 months ago
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/479b56b160d8
do not run Talos on nightly builds. r=rwood

Comment 10

5 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/479b56b160d8
Status: ASSIGNED → RESOLVED
Last Resolved: 5 months ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
You need to log in before you can comment on or make changes to this bug.