Open Bug 1520726 Opened 1 year ago Updated 2 months ago

Check fixes for perf regressions

Categories

(Tree Management :: Perfherder, enhancement, P3)

enhancement

Tracking

(Not tracked)

People

(Reporter: igoldan, Unassigned)

Details

(Keywords: feature, student-project)

A subset of the perf regressions bugs we file receive code fixes.
These must be verified by Perf sheriffs. What we normally do is go through each regression from the original alert summary and see whether the differences got reverted (not the percentages!). What's worse: Perfherder doesn't show this difference, we must mentally compute it.

Here's a real world example:

We need to automate this checking. By providing the ids of 2 alert summaries, Perfherder should be able to tell us whether the 2nd summary has canceled the regressions from the 1st summary. It should do so by comparing the differences with some level of approximation.

To avoid any kind of confusion, I should point to what this difference actually is. When an alert lands, it mentions the original baseline and the new baseline. The difference I'm talking about is the one between these 2 baselines.
If a regression increases a baseline by +200 milliseconds, a confirmed fix should find an approximately -200 milliseconds change.

Keywords: good-first-bug
No longer blocks: 1520720
You need to log in before you can comment on or make changes to this bug.