Open Bug 1263185 Opened 4 years ago Updated 3 years ago

Create an algorithm to get fastest and most frequently failing tests from ActiveData

Categories

(Testing :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

People

(Reporter: ahal, Unassigned)

References

(Blocks 1 open bug)

Details

For a given suite, we want an algorithm that returns a list of tests that are both fast, and fail frequently. This can be done with ActiveData. We also want to exclude known intermittents from the failure rates, which could potentially be done with autostar (though I'm not sure how easy getting the failure strings out of ActiveData is).

It's not really clear how important speed vs failure rates are or what magic mix of the two will give optimal results. So there should be some way of tweaking parameters and thresholds to make experimentation easy.

The script at testing/runtimes/writeruntimes.py could be a useful starting point to copy for this work.
You need to log in before you can comment on or make changes to this bug.