1583284 - [meta] Validate --browsertime Raptor tasks against vanilla Raptor harness tasks

Status: RESOLVED FIXED

(Testing :: Performance, task, P1)

Description

[Nick Alexander :nalexander [he/him]]

This is a meta ticket tracking validating the --browsertime Raptor tasks against the vanilla Raptor harness tasks. That is, we need to be confident that the data we get from Browsertime has similar or preferable population statistics as the data we get from Raptor tasks right now.

I think we want:

1. the two populations to have similar means. "In the ballpark" is good enough here.
2. the Browsertime population to have lower variance than the Raptor population.

Comment 1

[Nick Alexander :nalexander [he/him]]

This probably depends on warmloads, or we will all go nuts waiting for local test runs to complete: Bug 1577905.

Comment 2

[Robert Wood [:rwood]]

:sparky, do you have a different bug / duplicate that you've been using for the validation work? If so please close this one as a duplicate of yours, thanks!

Comment 3

[Greg Mierzwinski [:sparky]]

I've been adding everything to the following doc so this bug is not a duplicate.

For everyone interested, our findings can be found here: https://docs.google.com/document/d/1qCCWdlzQZSVzi1d7SKm8ReyJKX5KzdTnqMkHN59Cd9w/edit?usp=sharing

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

:sparky is there anything else to do here?

Comment 5

[Greg Mierzwinski [:sparky]]

Nothing left here, we are good to go for an android migration, and desktop issues are investigated in this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1610389