Open Bug 1309250 Opened 3 years ago Updated Last year

[moztest] StructuredTestRunner.run() does not always return test results

Categories

(Testing :: Mozbase, defect, P3)

Version 3
defect

Tracking

(Not tracked)

People

(Reporter: whimboo, Unassigned)

References

Details

In case of an exception when calling "test(result)", the finally block will correctly stop the test run but doesn't add the stop time nor return a valid test result:

https://dxr.mozilla.org/mozilla-central/rev/7be6b348c431d69f96f0765af3a0c0a0fe56d4bf/testing/mozbase/moztest/moztest/adapters/unit.py#199

This raises a problem for customers of this code because they are missing the result of this test. I have seen this in Marionette:

https://dxr.mozilla.org/mozilla-central/rev/7be6b348c431d69f96f0765af3a0c0a0fe56d4bf/testing/marionette/harness/marionette/runner/base.py#1064

An exception as passed through will cause the full code block (lines 1065 - 1086) not to be run. So the final printed results are wrong.

James, any idea who we can improve that? Maybe we attach the result to the exception which we could raise again after collecting all result information? That way we could make sure that customers can actually get the information.
Flags: needinfo?(james)
I had a look at the unittest.TextTestRunner class and it actually behaves the same. So it means that the `test()` method should always catch all exceptions? If not then we definitely would loose test results for all test methods contained in that test suite.
So it is unclear to me what the exact issue here is (this code was probably just copied from unittest in the first place). If you run a test and it throws an unexpected exception, what are you seeing vs what you expect to see?
Flags: needinfo?(james)
Priority: -- → P3
You need to log in before you can comment on or make changes to this bug.