1768861 - Investigate variance on perf try runs after switching to replicates

Status: NEW

(Testing :: Raptor, task, P3)

Description

[Greg Mierzwinski [:sparky]]

It's currently possible that the medians of the replicates is causing issues with observed variance over time. For instance, here are two try runs done days apart on the same parent commit that show a small but significant change in the metrics: https://treeherder.mozilla.org/perfherder/compare?originalProject=try&originalRevision=486587fc1b548687ebb61f56cc7e9f12d7d3f266&newProject=try&newRevision=173892be2336c5d78a844865c8f2c71ea58e205f&framework=13&page=1&showOnlyImportant=1&pageTitle=%5Bplacebo%5D+-ftrivial-auto-var-init%3Dpattern

After we switch to using replicates instead of medians, we should check to see if this issue still exists.