1263185 - Create an algorithm to get fastest and most frequently failing tests from ActiveData

Status: NEW

(Testing :: General, defect)

Description

[Andrew Halberstadt [:ahal]]

For a given suite, we want an algorithm that returns a list of tests that are both fast, and fail frequently. This can be done with ActiveData. We also want to exclude known intermittents from the failure rates, which could potentially be done with autostar (though I'm not sure how easy getting the failure strings out of ActiveData is).

It's not really clear how important speed vs failure rates are or what magic mix of the two will give optimal results. So there should be some way of tweaking parameters and thresholds to make experimentation easy.

The script at testing/runtimes/writeruntimes.py could be a useful starting point to copy for this work.