Create an algorithm to get fastest and most frequently failing tests from ActiveData

NEW
Unassigned

Status

Testing
General
a year ago
a year ago

People

(Reporter: ahal, Unassigned)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

a year ago
For a given suite, we want an algorithm that returns a list of tests that are both fast, and fail frequently. This can be done with ActiveData. We also want to exclude known intermittents from the failure rates, which could potentially be done with autostar (though I'm not sure how easy getting the failure strings out of ActiveData is).

It's not really clear how important speed vs failure rates are or what magic mix of the two will give optimal results. So there should be some way of tweaking parameters and thresholds to make experimentation easy.

The script at testing/runtimes/writeruntimes.py could be a useful starting point to copy for this work.
You need to log in before you can comment on or make changes to this bug.