1170144 - investigate run-by-dir cycle times as the harness/browser startup/shutdown takes a long time

Status: RESOLVED INACTIVE

(Testing :: Mochitest, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we run browser-chrome/devtools tests in --run-by-dir mode.  This is great for test isolation.  

While working towards bringing this to other tests (mochitest-plain/mochitest-chrome), I see we are doubling our runtimes.  running per dir gives us much more advantages than not (think simplicity for chunking, parallelization, test isolation).  

In this bug we should investigate and implement methods for now to make the harness startup/shutdown faster.  We can do stuff in parallel (not run the tests, but the rest of the work), or just optimize our path for simplicity.

Comment 1

[(not currently active) Ted Mielczarek]

We talked about this a bit in PDX. I think the first step would be to instrument things to find out what exactly is taking the most time between runs. If it's mostly browser startup then obviously that's a tricky thing to deal with. We'd have to start a browser in the background while the previous chunk is still running and then only start running tests in it when the previous tests finish.

If we're spending a lot of time doing non-startup things like creating the profile then we can certainly cut that down without complicating things much.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

for example in mochitest-other with run-by-dir we launch and terminate the browser 96 times:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d20914fd2b64

without it, we do it 3 times.  This takes us from 75 to 105 minutes- adding a leak, timeout, etc. we cross the 120 minute threshold.

93 extra jobs adds 30 minutes, otherwise we add 20 seconds for startup/shutdown.

possibly we could batch directories with <4 test files together to reduce the number of chunks we run.