1519507 - Allow running tests on multiple trees in parallel

Status: NEW

(Testing :: General, defect, P3)

Description

[Botond Ballo [:botond]]

Often I work with multiple trees on the same machine, and while e.g. I'm waiting for a test run on one tree, I'd like to do something else on another tree. Sometimes that "something else" is itself a test run.

For example, right now I have a tree with a desktop build and a tree with an Android build, and I'm running |mach reftest <filename>| in my Android build, and I tried to run |mach wpt <filename>| in my desktop build but got:

EnvironmentError: Servers failed to start: ws:8888

File "/home/botond/dev/mozilla/central/testing/web-platform/mach_commands.py", line 380, in run_wpt
return self.run_web_platform_tests(**params)
File "/home/botond/dev/mozilla/central/testing/web-platform/mach_commands.py", line 372, in run_web_platform_tests
return wpt_runner.run(logger, **params)
File "/home/botond/dev/mozilla/central/testing/web-platform/mach_commands_base.py", line 40, in run
result = wptrunner.start(**kwargs)
File "/home/botond/dev/mozilla/central/testing/web-platform/tests/tools/wptrunner/wptrunner/wptrunner.py", line 339, in start
return not run_tests(**kwargs)
File "/home/botond/dev/mozilla/central/testing/web-platform/tests/tools/wptrunner/wptrunner/wptrunner.py", line 196, in run_tests
test_environment.ensure_started()
File "/home/botond/dev/mozilla/central/testing/web-platform/tests/tools/wptrunner/wptrunner/environment.py", line 207, in ensure_started
", ".join("%s:%s" % item for item in failed))

It would be nice if doing multiple local test runs like this was supported.

Comment 1

[Geoff Brown [:gbrown]]

Some test suites have command line options to specify the server ports; using those so that each run is using a distinct server might enable this scenario.