Make pytest-mozlog compatible with pytest-xdist

RESOLVED FIXED

Status

RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: vakila, Unassigned, Mentored)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
The pytest-xdist plugin [1] allows for tests to be run in parallel, which we should take advantage of in the Marionette harness tests as it could speed up the test runs. 

However, there are some compatibility issues between the way the harness tests use Pytest and the pytest_mozlog plugin (see Bug 1285299) that need to be resolved, such as:

- pytest-xdist appears to rely on the terminalreporter plugin being enabled, whereas the Mn-h tests disable this plugin to replace Pytest's terminal output 

- the `TestReport` objects created by pytest-xdist are slightly different than those of vanilla pytest. For example, the report's `longrepr` is of a different format, which breaks some of the pytest_mozlog reporting hooks

- with the plugin enabled, certain hooks (e.g. `pytest_collection_modifyitems`) at different times or with different frequencies than with vanilla Pytest, which could also affect pytest_mozlog


There may also be further compatibility issues, these are just a few initially apparent ones. These issues should be investigated and the pytest_mozlog plugin/its invocation in the Mn-h tests should be made compatible with xdist. 


[1] https://pypi.python.org/pypi/pytest-xdist
This is likely to block the web automation from providing suitable log formats to Treeherder due to the heavy use of pytest-xdist. For now I'll put myself down as a mentor for anyone that might like to look into this, as there are no immediate plans for web automation to start pushing results to Treeherder.
Mentor: dave.hunt
Changing the summary/component to reflect that Marionette Harness Tests don't really need pytest-xdist. (These tests are very short and I expect them to remain that way, so parallelization isn't a priority.) Of course, pytest-xdist support is desirable and we should make changes to Mn-h invocation if that helps.

Re terminalreporter, I think it's not crucial for Mn-h output on Treeherder to have it disabled: if it's enabled while we use the -qq command-line option with pytest, we get barely any pytest output at all, which is fine.
Component: Marionette → Mozbase
Summary: Use pytest-xdist for parallelization of Marionette harness tests → Make pytest-mozlog compatible with pytest-xdist
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Depends on: 1338531, 1338534
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.