Closed Bug 476102 Opened 12 years ago Closed 12 years ago

auto-detect variance in talos results

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: joduinn, Assigned: catlee)

Details

Until now, we rely on manual inspection to notice abnormal variance in results from talos machines. So, a human would look on graph server to see if the three machines reporting perf results for a given project branch are all within some "gutcheck" % deviation from each other.  Now, with auto-rebooting, results seem more stable, and the variance seems much reduced. 

Is there any way to check for possible variance programatically? 

Apart from being more human-friendly, this might also enable us to automatically warning people by email/tinderbox of large changes in results.
Moving into catlee's queue as he's working on the tool that automates this analysis.
Assignee: nobody → catlee
Component: Release Engineering: Future → Release Engineering
The analysis code is checked into the graph server repo at:
http://hg.mozilla.org/graphs/server/analysis

And is currently running via a cronjob on cruncher.build.mozilla.org.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.