1291486 - offer a way to not rerun (xpcshell) test failures (and/or inspect the output from such failed tests)

Status: RESOLVED FIXED

(Testing :: XPCShell Harness, defect, P3)

Description

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

If I run `./mach xpcshell-test security/manager/ssl/tests/unit/`, test_pkcs11_insert_remove.js will consistently fail when run in parallel, but it will always succeed when rerun by mach (or xpcshell - I'm not actually sure who's doing what here). I would like to know why it failed, but mach (or xpcshell - again I'm not sure who's in charge here) doesn't give me any informative output (because, after all, everything eventually succeeded). I guess either I want a way to look at the output from the original failing test or to have mach/xpcshell treat that failure as a failure and dump the output from it.

Comment 1

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Oh, I guess there's '--verbose', but that gives me way too much output - I really only want the output from the failing test.

Comment 2

[Mark Banner (:standard8)]

I think I know a way of doing this, just have to figure out how to get an appropriate argument into it.

How about --verbose-if-fails ?

Comment 3

[Mark Banner (:standard8)]

Attachment: Bug 1291486 - Add an option (--verbose-if-fails) to mach's xpcshell-test to output test log if it fails when it will be retried. r?ahal

When running xpcshell-tests, if a test fails when run in parallel, and then passes when re-run sequentially, the log cannot be seen. Verbose mode is not good enough as that interweaves all the logs.

This adds an option that allows printing only the log of the test that fails.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3565a43b5387
Add an option (--verbose-if-fails) to mach's xpcshell-test to output test log if it fails when it will be retried. r=ahal

Comment 5

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/3565a43b5387