auto-detect variance in talos results

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
8 years ago
4 years ago

People

(Reporter: joduinn, Assigned: catlee)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Until now, we rely on manual inspection to notice abnormal variance in results from talos machines. So, a human would look on graph server to see if the three machines reporting perf results for a given project branch are all within some "gutcheck" % deviation from each other.  Now, with auto-rebooting, results seem more stable, and the variance seems much reduced. 

Is there any way to check for possible variance programatically? 

Apart from being more human-friendly, this might also enable us to automatically warning people by email/tinderbox of large changes in results.

Comment 1

8 years ago
http://atlee.ca/blog/2009/02/05/automated-talos-analysis/
Moving into catlee's queue as he's working on the tool that automates this analysis.
Assignee: nobody → catlee
Component: Release Engineering: Future → Release Engineering
(Assignee)

Comment 3

8 years ago
The analysis code is checked into the graph server repo at:
http://hg.mozilla.org/graphs/server/analysis

And is currently running via a cronjob on cruncher.build.mozilla.org.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.