Open Bug 858197 Opened 7 years ago Updated 7 years ago

Python API for receiving test progress

Categories

(Testing :: Mochitest, defect)

defect
Not set

Tracking

(Not tracked)

People

(Reporter: gps, Unassigned)

Details

Currently the mochitest runtests.py API returns a status code. All detail about running tests is contained in the log output.

This bug is a hand-wavy feature request for the Python test runner to expose - via Python - details about individual test results.

The scenario I want to enable is for tools built on top of the test runners to be able to inspect and react to test results in their own special way. e.g. the thing running the tests may want to print the list of tests that failed at the end of execution.

The two high-level options to provide this are "return value" and "callback function." The latter provides opportunities for a lot more control and thus is the one I'd like to see. e.g. the "on test result" callback could be like "oh, this is an interactive test session, let me hook up a debugger!" mach --debug-on-failure anyone?

It's also worth noting that by exposing a callback-driven "test results and control" API to the Python test runner, you move the burden of implementing additional test running features to tools built on top of the test runner rather than inside the test runner itself. e.g. all this xunit code we wrote (see bug 725478) wouldn't have to live in the test runners and the test runners could focus on the job they do best: running tests. That might open the door to a more unified test controller that offers common result formatting, etc across test suites. Just a thought.
Piling on that this is a good idea, and that the mach |python-test| command implemented in Bug 818744 is an example of such a test runner.
You need to log in before you can comment on or make changes to this bug.