706368 - Add option to stop running tests on first error.

Status: RESOLVED FIXED

(Add-on SDK Graveyard :: General, defect, P2)

Description

[Mark Hammond [:markh] [:mhammond]]

If running a test suite that takes a long time to complete, it can be very useful to have the test run stop as soon as the first error is encountered for 2 main reasons:

* The error relating to the failed test is much easier to find as it will be near the bottom of the output.

* In some cases the same simple error will result in many tests failing which generates a huge amount of noise.

I'm attaching a pull request which adds a new --stop-on-error option - if it is provided, the test run stops as soon as a test reports a failure.

Comment 1

[Mark Hammond [:markh] [:mhammond]]

Attachment: Pointer to Github pull request: https://github.com/mozilla/addon-sdk/pull/279

Pointer to Github pull-request

Comment 2

[Myk Melez [:myk] [@mykmelez]]

Comment on attachment 577848 [details]
Pointer to Github pull request: https://github.com/mozilla/addon-sdk/pull/279

r=myk, but see comments about unexpected behavior in pull request.  The current behavior when a test fails on a `cfx testex` or `cfx testpkgs` run doesn't seem quite right.

Comment 3

[Mark Hammond [:markh] [:mhammond]]

Comment on attachment 577848 [details]
Pointer to Github pull request: https://github.com/mozilla/addon-sdk/pull/279

There are some changes to the output generated by some of the 'test' commands - eg:

* if 'testex' is used and an "example" test fails, the output is now:

Some examples tests were unsuccessful.

(no "summary" line was printed at all before - similarly for 'testpkgs')

* if 'testall' is used and an "example" test fails, the output is now:

Some examples tests were unsuccessful.
Some tests were unsuccessful.

(the 2nd-last line is new and while one of the lines seems redundant, it is the easiest way to have the single line printed for 'testex', as shown above)

Primarily for this reason I'm re-requesting review.

Comment 4

[[github robot]]

Commit pushed to https://github.com/mozilla/addon-sdk

https://github.com/mozilla/addon-sdk/commit/92907ff34b2d95ac5e1bac101dd05182c950655b
Merge pull request #279 from mhammond/bug/706368-stop-on-error

bug 706368 - add --stop-on-error option to make tests stop after first failure; r=@mykmelez