Closed Bug 801679 Opened 7 years ago Closed 4 years ago

mozunit doesn't handle expected failures

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(firefox47 fixed)

RESOLVED FIXED
mozilla47
Tracking Status
firefox47 --- fixed

People

(Reporter: glandium, Unassigned)

References

Details

Attachments

(1 file)

python's unittest comes with a decorator for expected failures, but mozunit doesn't handle them.
Example test:

import mozunit, unittest

class Test(unittest.TestCase):
    @unittest.expectedFailure
    def test_fail(self):
        self.assertEqual(1, 0)

    @unittest.expectedFailure
    def test_pass(self):
        self.assertEqual(0, 0)

    def test_normal(self):
        self.assertEqual(0, 0)

mozunit.main()

Result:
TEST-PASS | /tmp/test.py | test_normal

Expected result:
TEST-KNOWN-FAIL | /tmp/test.py | test_fail
TEST-UNEXPECTED-SUCCESS | /tmp/test.py | test_pass
TEST-PASS | /tmp/test.py | test_normal

or something like that.

For reference, unittest.main() outputs:
x.u
----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK (expected failures=1, unexpected successes=1)
Heh, I was about to file the same bug again... thanks bugzilla search during filing...
Blocks: 1247833
Python's unittest has decorators to mark tests as skipped or expecting
to fail. Mozunit runner fails to print anything for them.

While here, somehow unify the output for TEST-UNEXPECTED-FAIL.

This leaves known failures silent about the failure. It would be better
to only show them with a flag, like -v, and to leave it to a followup.

Review commit: https://reviewboard.mozilla.org/r/34709/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/34709/
Attachment #8718693 - Flags: review?(gps)
Attachment #8718693 - Flags: review?(gps) → review+
Comment on attachment 8718693 [details]
MozReview Request: Bug 801679 - Handle expected failures and skipped tests in mozunit runner. r=gps

https://reviewboard.mozilla.org/r/34709/#review31681

Are there no tests for mozunit that could be updated?
(In reply to Gregory Szorc [:gps] from comment #4)
> Are there no tests for mozunit that could be updated?

There's only a test for MockOpen.
https://hg.mozilla.org/mozilla-central/rev/199a9500951d
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla47
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.