1520726 - Check fixes for perf regressions

Status: NEW

(Tree Management :: Perfherder, enhancement, P3)

Description

[Ionuț Goldan [:igoldan]]

A subset of the perf regressions bugs we file receive code fixes.
These must be verified by Perf sheriffs. What we normally do is go through each regression from the original alert summary and see whether the differences got reverted (not the percentages!). What's worse: Perfherder doesn't show this difference, we must mentally compute it.

Here's a real world example:

• original alert summary with regressions
• alert that showed up after the code fix landed

We need to automate this checking. By providing the ids of 2 alert summaries, Perfherder should be able to tell us whether the 2nd summary has canceled the regressions from the 1st summary. It should do so by comparing the differences with some level of approximation.

To avoid any kind of confusion, I should point to what this difference actually is. When an alert lands, it mentions the original baseline and the new baseline. The difference I'm talking about is the one between these 2 baselines.
If a regression increases a baseline by +200 milliseconds, a confirmed fix should find an approximately -200 milliseconds change.