Closed Bug 669445 Opened 11 years ago Closed 6 years ago

Write a tool to do regression searching on intermittent test failures

Categories

(Testing :: General, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: ted, Unassigned)

References

Details

Regression searching is extremely valuable to find out where something broke. With intermittent test failures, it's harder because the test may or may not fail in any given build. We might be able to overcome this by running the tests multiple times in a row at every step, it needs investigation to see if it's a workable strategy.
See Also: → orangeseed
Assignee: nobody → mdas
Depends on: 603517
Proposed Solution
----------
mozcommitbuilder (github.com/samliu/mozcommitbuilder) is a regression finder that bisects, builds, and tests commits. It has support for inputting a special script as the "revision is good/bad" condition.

Goal: Write a condition script that uses the looping test runner [dependency] to run given test(s) x times (e.g 1000) instead of just once to find nondeterministic regressions on a given range. The condition function should return "bad" for either a single failure or a given threshold of failure (e.g if failure rate is higher than 1%?)
No longer depends on: 603517
Assignee: mdas → nobody
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.