Closed Bug 669445 Opened 11 years ago Closed 6 years ago
Write a tool to do regression searching on intermittent test failures
Regression searching is extremely valuable to find out where something broke. With intermittent test failures, it's harder because the test may or may not fail in any given build. We might be able to overcome this by running the tests multiple times in a row at every step, it needs investigation to see if it's a workable strategy.
Assignee: nobody → mdas
Depends on: 603517
Proposed Solution ---------- mozcommitbuilder (github.com/samliu/mozcommitbuilder) is a regression finder that bisects, builds, and tests commits. It has support for inputting a special script as the "revision is good/bad" condition. Goal: Write a condition script that uses the looping test runner [dependency] to run given test(s) x times (e.g 1000) instead of just once to find nondeterministic regressions on a given range. The condition function should return "bad" for either a single failure or a given threshold of failure (e.g if failure rate is higher than 1%?)
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.