858197 - Python API for receiving test progress

Status: NEW

(Testing :: Mochitest, defect)

Description

[Gregory Szorc [:gps]]

Currently the mochitest runtests.py API returns a status code. All detail about running tests is contained in the log output.

This bug is a hand-wavy feature request for the Python test runner to expose - via Python - details about individual test results.

The scenario I want to enable is for tools built on top of the test runners to be able to inspect and react to test results in their own special way. e.g. the thing running the tests may want to print the list of tests that failed at the end of execution.

The two high-level options to provide this are "return value" and "callback function." The latter provides opportunities for a lot more control and thus is the one I'd like to see. e.g. the "on test result" callback could be like "oh, this is an interactive test session, let me hook up a debugger!" mach --debug-on-failure anyone?

It's also worth noting that by exposing a callback-driven "test results and control" API to the Python test runner, you move the burden of implementing additional test running features to tools built on top of the test runner rather than inside the test runner itself. e.g. all this xunit code we wrote (see bug 725478) wouldn't have to live in the test runners and the test runners could focus on the job they do best: running tests. That might open the door to a more unified test controller that offers common result formatting, etc across test suites. Just a thought.

Comment 1

[Nick Alexander :nalexander [he/him]]

Piling on that this is a good idea, and that the mach |python-test| command implemented in Bug 818744 is an example of such a test runner.