1301494 - Make pytest-mozlog compatible with pytest-xdist

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Anjana Vakil (:vakila)]

The pytest-xdist plugin [1] allows for tests to be run in parallel, which we should take advantage of in the Marionette harness tests as it could speed up the test runs. 

However, there are some compatibility issues between the way the harness tests use Pytest and the pytest_mozlog plugin (see Bug 1285299) that need to be resolved, such as:

- pytest-xdist appears to rely on the terminalreporter plugin being enabled, whereas the Mn-h tests disable this plugin to replace Pytest's terminal output 

- the `TestReport` objects created by pytest-xdist are slightly different than those of vanilla pytest. For example, the report's `longrepr` is of a different format, which breaks some of the pytest_mozlog reporting hooks

- with the plugin enabled, certain hooks (e.g. `pytest_collection_modifyitems`) at different times or with different frequencies than with vanilla Pytest, which could also affect pytest_mozlog


There may also be further compatibility issues, these are just a few initially apparent ones. These issues should be investigated and the pytest_mozlog plugin/its invocation in the Mn-h tests should be made compatible with xdist. 


[1] https://pypi.python.org/pypi/pytest-xdist

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This is likely to block the web automation from providing suitable log formats to Treeherder due to the heavy use of pytest-xdist. For now I'll put myself down as a mentor for anyone that might like to look into this, as there are no immediate plans for web automation to start pushing results to Treeherder.

Comment 2

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Changing the summary/component to reflect that Marionette Harness Tests don't really need pytest-xdist. (These tests are very short and I expect them to remain that way, so parallelization isn't a priority.) Of course, pytest-xdist support is desirable and we should make changes to Mn-h invocation if that helps.

Re terminalreporter, I think it's not crucial for Mn-h output on Treeherder to have it disabled: if it's enabled while we use the -qq command-line option with pytest, we get barely any pytest output at all, which is fine.