1301177 - Marionette harness tests run locally with mach should have same logs as in automation

Status: NEW

(Testing :: Marionette Client and Harness, defect, P5)

Description

[Anjana Vakil (:vakila)]

Once Bug 1285299 lands, the logs for the Marionette harness tests job (Mn-h) on Taskcluster will be in Mozlog's TBPL format, thanks to the new pytest_mozlog Pytest plugin which is becoming a part of the Mozlog package.

However, to use this plugin when running the tests locally with `mach python-test`, we need to manually register the pytest_mozlog plugin with Pytest, e.g. via extra command-line arguments in a pytest.ini file or the `if __name__` blocks of each test module, or by adding a `pytest_plugins` line to conftest.py [1]. 

This is because of the way mach loads Python packages into the virtualenv [2], i.e. copying the modules into the path instead of installing them using their setup.py files. When the Mozlog package is installed with setup.py, as is the case in automation or if we populate a local virtualenv by hand to run the tests locally with `py.test`, Pytest finds the entry point defined in Mozlog's setup.py and registers the plugin. 

This creates a problem for mach compatibility, because if we try to register the plugin manually in the command-line args or conftest.py, Pytest complains that the plugin is already registered. This means we essentially need to have two different configurations for running the tests, one for automation/py.test and one for mach, which is not idea.

We should change the way the pytest_mozlog plugin is registered when running the tests with mach so that it does not require command-line arguments or `pytest_plugins`. Possible approaches include:
- Change the way mach installs the Mozlog package to use "setup.py" instead of "python.pth" [3]
- Use pytest hook(s) (e.g. `pytest_configure`) in the harness tests' conftest.py to register the pytest_mozlog plugin if not present (e.g. [4])


[1] http://doc.pytest.org/en/2.9.2/plugins.html#using-plugins

[2] https://dxr.mozilla.org/mozilla-central/rev/91c2b9d5c1354ca79e5b174591dbb03b32b15bbf/python/mozbuild/mozbuild/virtualenv.py

[3] https://dxr.mozilla.org/mozilla-central/rev/91c2b9d5c1354ca79e5b174591dbb03b32b15bbf/testing/mozbase/packages.txt#11

[4] http://doc.pytest.org/en/2.9.2/writing_plugins.html#optionally-using-hooks-from-3rd-party-plugins

Comment 1

[David Burns :automatedtester]

[mass update] Setting priority