Closed Bug 1583284 Opened 5 years ago Closed 4 years ago

[meta] Validate --browsertime Raptor tasks against vanilla Raptor harness tasks

Categories

(Testing :: Performance, task, P1)

Version 3
task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nalexander, Assigned: sparky)

References

(Blocks 1 open bug)

Details

(Keywords: meta)

This is a meta ticket tracking validating the --browsertime Raptor tasks against the vanilla Raptor harness tasks. That is, we need to be confident that the data we get from Browsertime has similar or preferable population statistics as the data we get from Raptor tasks right now.

I think we want:

  1. the two populations to have similar means. "In the ballpark" is good enough here.
  2. the Browsertime population to have lower variance than the Raptor population.

This probably depends on warmloads, or we will all go nuts waiting for local test runs to complete: Bug 1577905.

Depends on: 1577905
Summary: Validate --browsertime Raptor tasks against vanilla Raptor harness tasks → [meta] Validate --browsertime Raptor tasks against vanilla Raptor harness tasks
Depends on: 1584245
Depends on: 1581990

:sparky, do you have a different bug / duplicate that you've been using for the validation work? If so please close this one as a duplicate of yours, thanks!

Flags: needinfo?(gmierz2)

I've been adding everything to the following doc so this bug is not a duplicate.

For everyone interested, our findings can be found here: https://docs.google.com/document/d/1qCCWdlzQZSVzi1d7SKm8ReyJKX5KzdTnqMkHN59Cd9w/edit?usp=sharing

Flags: needinfo?(gmierz2)
Priority: -- → P1
Assignee: nobody → gmierz2
Status: NEW → ASSIGNED

:sparky is there anything else to do here?

Flags: needinfo?(gmierz2)

Nothing left here, we are good to go for an android migration, and desktop issues are investigated in this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1610389

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Flags: needinfo?(gmierz2)
Resolution: --- → FIXED
See Also: → 1610389
You need to log in before you can comment on or make changes to this bug.