1641623 - Add a numerical summary of Perfherder improvement/regression to the Compare Revisions view

Status: NEW

(Tree Management :: Perfherder, enhancement, P3)

Description

[Andrew Creskey [:acreskey]]

This bug is to discuss adding one or more numerical summaries to the Perfherder Compare Revisions view

When comparing the performance of different revisions against each other, the performance team has found it to be quite useful to look at an "overall number" for each metric.

For instance, in large-scale local pageload tests, we will look at the geomean of improvements, over a wide range of sites, for a given metric (e.g. load event).

An example can be seen here:
https://docs.google.com/spreadsheets/d/18qCiz3SReDgDPwhbYfuDrbnBK1030FuVWGBHWwdgCFY/edit#gid=1999584773&range=AC44

In this case, we see that this build of Fenix showed a ~31.42% overall improvement in pageload time relative to the baseline build.

We've found these summaries to be numerical quite stable - even if per-site results vary significantly, the overall improvements are consistently reproducible within a ~1-2% range.

I propose adding a similar calculated result to Perfherder Compare Revision views such as this one:

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=106c2e16d815b9c9bad49038c8ce28303189e0b6&newProject=try&newRevision=fb574be816f331cc9718483d8c3c236dc14df158&framework=13