1429583 - Investigate container tabs for running tests in parallel

Status: NEW

(Testing :: General, enhancement, P3)

Description

[Gregory Szorc [:gps]]

Had an interesting discussion at lunch today. I noticed that we are spending a lot of money running GPU-enabled EC2 instances. Mentioned that to bdahl since his work to enable headless Firefox could result in us significantly lowering our AWS bill by moving away from GPU-enabled instances. We were lamenting the general issue with only being able to run a single test within a process (even if we start aggressively executing tests in headless mode we'll likely still want to run them in graphics mode periodically as a sanity check). mrbkap made the (IMO fantastic) suggestion that we could leverage Firefox container tabs to enable concurrent test execution *within the same process*.

Essentially, the test harness would spin up multiple tabs. Each tab would get its own content process and container tab. For all intents and purposes, they would behave like separate Firefox instances because of process separation, cookie separation, etc. The test harness (running as chrome-privileged code) could dispatch tests to containers/tabs when the tab is idle.

Things to work out:

* Tests requiring focus, clipboard, new windows, and other global state. (I think there are efforts to tag tests with this metadata and make tests fail if they aren't annotated. If not, this is a tractable problem.)
* Enforcing separate processes per tab. mrbkap made it sound like there may need to be some kind of special API the test harness would need to use.
* Stability/appropriateness of container tabs for this use case.
* Dealing with the long tail of tests that have implicit run-time dependencies. I /think/ jmaher or someone once did a Try push where each test was executed in its own process and this teased out tests relying on state from a previous test. We would need to solve this class of test reliability problems before any concurrent execution could be used.
* Relative priority against mass running tests in headless mode. Headless mode is probably easier to roll out. And headless mode supports tests that require focus. But if we go all-in on headless mode, we still need to run tests in a graphical environment. So it still makes sense to support parallel test execution in a graphical environment.

I'm not sure if this bug is actionable. But it would probably be useful if mrbkap, tanvi, etc chimed in with technical feasibility expertise from the gecko layer.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Additional to that I wonder how the logging would work. We should not garble everything because each test logs the results in-between the others. Especially with Firefox default log output it could become not useful at all.

Comment 2

[Gregory Szorc [:gps]]

We have Python code starting the Firefox process. We have privileged JS inside Firefox coordinating the tests. If the appropriate "markers" are inserted into the log stream during test execution, I would think/hope it would be possible to associate log events to tests based on the PID those messages came from. Events coming from main/shared PID or threads could be more difficult to tease out. A mitigation for that is to run failing tests in isolation so there is a "clean" log for that failure.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

here is a summary of tests which cannot repeat themselves:
https://docs.google.com/spreadsheets/d/1OZp9xgGJ5NYy8kDRZaUoJsoQrL7wf2xRBtZ_akIk5-c/edit#gid=0

we also have bug 1419135 on file for running mochitest in parallel