801679 - mozunit doesn't handle expected failures

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Mike Hommey [:glandium]]

python's unittest comes with a decorator for expected failures, but mozunit doesn't handle them.

Comment 1

[Mike Hommey [:glandium]]

Example test:

import mozunit, unittest

class Test(unittest.TestCase):
    @unittest.expectedFailure
    def test_fail(self):
        self.assertEqual(1, 0)

    @unittest.expectedFailure
    def test_pass(self):
        self.assertEqual(0, 0)

    def test_normal(self):
        self.assertEqual(0, 0)

mozunit.main()

Result:
TEST-PASS | /tmp/test.py | test_normal

Expected result:
TEST-KNOWN-FAIL | /tmp/test.py | test_fail
TEST-UNEXPECTED-SUCCESS | /tmp/test.py | test_pass
TEST-PASS | /tmp/test.py | test_normal

or something like that.

For reference, unittest.main() outputs:
x.u
----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK (expected failures=1, unexpected successes=1)

Comment 2

[Mike Hommey [:glandium]]

Heh, I was about to file the same bug again... thanks bugzilla search during filing...

Comment 3

[Mike Hommey [:glandium]]

Attachment: MozReview Request: Bug 801679 - Handle expected failures and skipped tests in mozunit runner. r=gps

Python's unittest has decorators to mark tests as skipped or expecting
to fail. Mozunit runner fails to print anything for them.

While here, somehow unify the output for TEST-UNEXPECTED-FAIL.

This leaves known failures silent about the failure. It would be better
to only show them with a flag, like -v, and to leave it to a followup.

Review commit: https://reviewboard.mozilla.org/r/34709/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/34709/

Comment 4

[Gregory Szorc [:gps]]

Comment on attachment 8718693 [details]
MozReview Request: Bug 801679 - Handle expected failures and skipped tests in mozunit runner. r=gps

https://reviewboard.mozilla.org/r/34709/#review31681

Are there no tests for mozunit that could be updated?

Comment 5

[Mike Hommey [:glandium]]

(In reply to Gregory Szorc [:gps] from comment #4)
> Are there no tests for mozunit that could be updated?

There's only a test for MockOpen.

Comment 6

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/199a9500951d

Comment 7

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/199a9500951d