1191871 - Add option to mochitest to end test on first unexpected result, for ease of bisecting

Status: NEW

(Testing :: Mochitest, enhancement)

Description

[Aryeh Gregor (:ayg) (no longer with Mozilla)]

When I get failures on try, I would like to bisect the push to see which caused it.  I can provide a script to the bisect command that will compile the code and run the test, but that runs the whole test.  Some tests are large, and can take a number of seconds to run after the first failure has been encountered.  Then the browser has to be shut down before the command returns, which adds a number of extra seconds on my machine.

I would like to have an option, perhaps named --abort-on-failure, that immediately crashes the browser and returns an error code as soon as an unexpected result is encountered.

Comment 1

[Aryeh Gregor (:ayg) (no longer with Mozilla)]

Does this sound reasonable?  If so, any pointers on how to implement it?  I could write a patch if someone points me in the right direction.

Comment 2

[Andrew Halberstadt [:ahal]]

So to clarify, you're only running one test (by supplying a path) and that single test takes too long? Or do you mean the entire test run takes too long?

I guess that's reasonable, though if the former seems somewhat edge casey. You might be able to look at --debug-on-failure and/or --screenshot-on-fail to get an idea on how to implement it.

Also, there's a similar option for debug builds called --fatal-assertions (it's suppressed from the help message though):
https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/mochitest_options.py#253

Comment 3

[(not currently active) Ted Mielczarek]

This seems fine, I can see the utility. You'll probably want to implement this inside the JS part of the harness if you want to crash and get a stack immediately, you can probably call nsIDebug2.abort from JS (by way of SpecialPowers).

The simplest place to do this would be in SimpleTest._logResult:
https://dxr.mozilla.org/mozilla-central/rev/f2518b8a7b97b5bb477e94bc9131584007aac887/testing/mochitest/tests/SimpleTest/SimpleTest.js#346