Closed Bug 1398953 Opened 2 years ago Closed 2 years ago

Use runUntilFailure in test-verify

Categories

(Testing :: General, enhancement)

enhancement
Not set

Tracking

(firefox57 fixed)

RESOLVED FIXED
mozilla57
Tracking Status
firefox57 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

In cases like https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=e78342f09ee6e90b78dc098742f3b0bae3b9bc84&filter-searchStr=test-verify&filter-tier=1&filter-tier=2&filter-tier=3, the test being verified fails frequently, so when we run the test with --repeat, the same failure is hit time and time again, cluttering the test summary.

I think it will be better / tidier / easier to understand if we run with runUntilFailure.
We already do this for reftests:

https://dxr.mozilla.org/mozilla-central/rev/f9a5e9ed62103c84e4cde915f4d08f1ce71be83e/layout/tools/reftest/runreftest.py#437

I believe there is no such option for xpcshell tests.
Attachment #8906806 - Flags: review?(jmaher)
See Also: → 1394910
Comment on attachment 8906806 [details] [diff] [review]
set runUntilFailure for mochitests

Review of attachment 8906806 [details] [diff] [review]:
-----------------------------------------------------------------

ok, cool!  Do  we need to adjust any limits about time or number of iterations?
Attachment #8906806 - Flags: review?(jmaher) → review+
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #2)
> ok, cool!  Do  we need to adjust any limits about time or number of
> iterations?

I don't think so, at least, not now -- let's see how this works out.
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9ade9aba5dac
Set runUntilFailure in mochitest with --verify option; r=jmaher
https://hg.mozilla.org/mozilla-central/rev/9ade9aba5dac
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
You need to log in before you can comment on or make changes to this bug.