669445 - Write a tool to do regression searching on intermittent test failures

Status: RESOLVED INCOMPLETE

(Testing :: General, defect)

Description

[(not currently active) Ted Mielczarek]

Regression searching is extremely valuable to find out where something broke. With intermittent test failures, it's harder because the test may or may not fail in any given build. We might be able to overcome this by running the tests multiple times in a row at every step, it needs investigation to see if it's a workable strategy.

Comment 1

[Sam Liu]

Proposed Solution
----------
mozcommitbuilder (github.com/samliu/mozcommitbuilder) is a regression finder that bisects, builds, and tests commits. It has support for inputting a special script as the "revision is good/bad" condition.

Goal: Write a condition script that uses the looping test runner [dependency] to run given test(s) x times (e.g 1000) instead of just once to find nondeterministic regressions on a given range. The condition function should return "bad" for either a single failure or a given threshold of failure (e.g if failure rate is higher than 1%?)