It's currently possible that our the medians of the replicates is causing issues with observed variance over time. For instance, here are two try runs done days apart on the same parent commit that show a small but significant change in the metrics: https://treeherder.mozilla.org/perfherder/compare?originalProject=try&originalRevision=486587fc1b548687ebb61f56cc7e9f12d7d3f266&newProject=try&newRevision=173892be2336c5d78a844865c8f2c71ea58e205f&framework=13&page=1&showOnlyImportant=1&pageTitle=%5Bplacebo%5D+-ftrivial-auto-var-init%3Dpattern After we switch to using replicates instead of medians, we should check to see if this issue still exists.
Bug 1768861 Comment 0 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
It's currently possible that the medians of the replicates is causing issues with observed variance over time. For instance, here are two try runs done days apart on the same parent commit that show a small but significant change in the metrics: https://treeherder.mozilla.org/perfherder/compare?originalProject=try&originalRevision=486587fc1b548687ebb61f56cc7e9f12d7d3f266&newProject=try&newRevision=173892be2336c5d78a844865c8f2c71ea58e205f&framework=13&page=1&showOnlyImportant=1&pageTitle=%5Bplacebo%5D+-ftrivial-auto-var-init%3Dpattern After we switch to using replicates instead of medians, we should check to see if this issue still exists.