1608454 - While running tests, don't wait for the previous Firefox instance to shutdown before starting a new one

Status: NEW

(Testing :: General, enhancement, P3)

Description

[Marco Castelluccio [:marco]]

We restart Firefox whenever we execute a new manifest in a task.
We are currently waiting for the process to shutdown before starting a new one. We could instead start the new one without waiting for the first to finish.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

We have thought about this in the past, a similar idea is to prewarm up the next profile and browser so when the current tests are done we start right off running tests while we cleanup.

A few quirks to consider:

• some tools/tests will use window manager api or process lists to determine firefox processes- we would need to spend some time supporting >1 firefox process (we have tools to set focus between tests, etc.)
• many tests send keyboard/mouse input, we would need to ensure the browser is in the foreground
• with multiple windows open, typically they are stacked and not fully overlapping, many tests depend on window position (for the mouse clicks)
• the logs would be interleaved (not sequential) and determining failures could be misleading- we would need log buffering and coalescing- I am mostly thinking about failure to shutdown, leak on shutdown.
• we could have resource locking issues (such as a log file we are trying to write to, create, etc.)

For reference, I believe :froydnj was working on making reftests run in parallel a few years back- they do not do the keyboard/mouse moves, nor check for open windows and focus as much, but a lot of the logging and other timing issues would have been looked at.

Additionally for xpcshell (not a browser window) we run in parallel (last 8 years) and if there is a failure we queue them up and run them sequentially at the end of the current run- we only report failures if it fails again in sequential mode.

Comment 2

[Nathan Froyd [:froydnj]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #1)

For reference, I believe :froydnj was working on making reftests run in parallel a few years back- they do not do the keyboard/mouse moves, nor check for open windows and focus as much, but a lot of the logging and other timing issues would have been looked at.

I believe all that code is still there; the main idea was that reftests (in the manifest, I think?) actually specify whether they need focus in some way, so we could run all the focus-requiring tests in a "chunk" and split up everything else into other "chunks" on the same machine.

It never got turned on in automation due to difficulties in figuring out some hangs, IIRC, but maybe with automation changes over the years, we could try again.

I don't know if we could easily retrofit focus-requirements onto mochitests (or WPT!); it seems...hard.

Comment 3

[Florian Quèze [:florian]]

Could we make shutdown much faster (ie. _exit(0) at the beginning of the shutdown code path) for test cases? Most of what happens during shutdown is writing data to the profile directory (should be possible to skip if we'll remove the profile after shutting down) and sending telemetry data (which I assume doesn't happen during tests).

Comment 4

[James Graham [:jgraham]]

I already implemented a simple version of this for wpt, which cut the total test time by ~10% in my measurements. Fortunately wpt didn't trigger the corner cases mentioned here, but certianly other harnesses are doing more things that depend on the details of system state.

For opt builds making shutdown faster could help; for debug builds the leak checking stuff all happens at shutdown and we can't skip that.