1258957 - log the reason for a test skip

Status: RESOLVED FIXED

(MozReview Graveyard :: Testing / Development Environment, defect)

Description

[:glob ✱]

if the docker environment is unable to start that test is skipped, however all diagnostic information is silently lost.

we should write the mozreview output to an .err file.

Comment 1

[:glob ✱]

Attachment: MozReview Request: testing: log mozreview output when a test is skipped (bug 1258957) r?gps

/dev/null is hard to inspect in order to determine why a test is skipped.  Log
to the test's .err file instead.

Review commit: https://reviewboard.mozilla.org/r/41881/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/41881/

Comment 2

[Gregory Szorc [:gps]]

Comment on attachment 8733713 [details]
MozReview Request: testing: log mozreview output when a test is skipped (bug 1258957) r?gps

https://reviewboard.mozilla.org/r/41881/#review38577

::: pylib/mercurial-support/run-tests.py:787
(Diff revision 1)
>                             + ' -c "import sys; sys.exit(0)"')
>          env["HGMERGE"] = "internal:merge"
>          env["HGUSER"]   = "test"
>          env["HGENCODING"] = "ascii"
>          env["HGENCODINGMODE"] = "strict"
> +        env["ERR_FILE"] = self.errpath

Not sure why you need this. I'd just redirect to $TESTMP/mozreview_start.log or something.

Comment 3

[:glob ✱]

https://reviewboard.mozilla.org/r/41881/#review38577

> Not sure why you need this. I'd just redirect to $TESTMP/mozreview_start.log or something.

logging into the same file where test errors live makes finding issues easier, especially in the case where multiple tests are run and they are skipped for different reasons (ie. transient 'mozreview start' failures).  reusing the existing err file removes any ambiguity about which test failed and why.

Comment 4

[:glob ✱]

https://hg.mozilla.org/hgcustom/version-control-tools/rev/91f4a0d24c62

Comment 5

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/41881/#review38577

> logging into the same file where test errors live makes finding issues easier, especially in the case where multiple tests are run and they are skipped for different reasons (ie. transient 'mozreview start' failures).  reusing the existing err file removes any ambiguity about which test failed and why.

$TESTTMP is a unique directory per test. You are echoing the contents of the failure, so no data is lost. There is also a --keep-tmpdir option or something like that will preserve the temporary test directories after execution.

Comment 6

[:glob ✱]

https://reviewboard.mozilla.org/r/41881/#review38577

> $TESTTMP is a unique directory per test. You are echoing the contents of the failure, so no data is lost. There is also a --keep-tmpdir option or something like that will preserve the temporary test directories after execution.

--keep-tmp is missing from the --help output for run-tests, and i think the default behavour should be to keep/display error messages.
ie. there's no need to make extra work for the dev when something fails.