1003401 - Mozbase unit test logs should include test name in failures.

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Dan Minor [:dminor]]

Currently failures look like:
13:46:58     INFO -  OSError: [Errno 2] No such file or directory

The desired output is
TEST-UNEXPECTED-FAIL | testname | message

See: https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#6.29_Outputs_failures_in_a_TBPL-starrable_format

Comment 1

[Dan Minor [:dminor]]

If you look at the current code [1] we are instantiating a unittest.TextTestRunner instance to run the tests.

As far as I can tell, the easiest way to change the error text would be to subclass unittest.TextTestResult as TBPLTextTestResult and change the error reporting there. You could then call the TextTestRunner constructor with resultclass=TBPLTextTestResult.

You can browse the source for the python unittest library at [2].

[1] http://mxr.mozilla.org/mozilla-central/source/testing/mozbase/test.py#88
[2] http://hg.python.org/cpython/file/33a39dfc239e/Lib/unittest/runner.py

Comment 2

[Julien Pagès (:parkouss)]

Attachment: Mozbase unit test logs should include test name in failures

The output of running python test.py is now something like:

test_parentheses (test_expressionparser.ExpressionParserTest) ... ok
test_comments (test_manifestparser.TestManifestParser) ... 
TEST-UNEXPECTED-FAIL | test_manifestparser.TestManifestParser.test_comments | False is not true
test_copy (test_manifestparser.TestManifestParser)
Test our ability to copy a set of manifests ... ok

Is that ok ?

Comment 3

[Dan Minor [:dminor]]

Comment on attachment 8426080 [details] [diff] [review]
Mozbase unit test logs should include test name in failures

Review of attachment 8426080 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks, one small fix, but otherwise looks good!

::: testing/mozbase/test.py
@@ +39,5 @@
> +        super(unittest.TextTestResult, self).addUnexpectedSuccess(test)
> +        msg = "unexpected pass"
> +        self.stream.writeln()
> +        self.stream.writeln('TEST-UNEXPECTED-PASS | %s | %s' % (test.id(), msg))
> +

Instead of using a separate msg variable, you might as well do:
self.stream.writeln('TEST-UNEXPECTED-PASS | %s | Unexpected pass' % test.id())

Comment 4

[Julien Pagès (:parkouss)]

Attachment: Mozbase unit test logs should include test name in failures

Sure - done ;)

I thought maybe short lines would be better; I'm used to 80 characters, as described in PEP 8, but I see that it is not the case in python mozilla's code. Do you have a constraint on line length, or maybe a coding style guide that I can see ?

Thanks for the time !

Comment 5

[Dan Minor [:dminor]]

Comment on attachment 8426414 [details] [diff] [review]
Mozbase unit test logs should include test name in failures

Review of attachment 8426414 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good and tested well locally with some injected failures. Thanks for your help on this!

Comment 6

[Dan Minor [:dminor]]

Try run here: https://tbpl.mozilla.org/?tree=Try&rev=b23b4516b8cc

Comment 7

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/1877d8afd103

and also thanks for contributing to Mozilla!

Comment 8

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/1877d8afd103