1615537 - Add an option to repeat a set of tests sequentially

Status: NEW

(Testing :: Mochitest, enhancement, P3)

Description

[Nicolas Chevobbe [:nchevobbe]]

Sometimes, I have test failures (e.g. in b.js) because a previous test (a.js) had an unexpected impact on the failing test. In such case, I want to be able to run those 2 tests one after the other, eventually multiple time so I can reproduce an intermittent failure (and then assert it's fixed).

Today, if I run ./mach mochitest a.js b.js --repeat 3, the tests will run in this order

a.js
a.js
a.js
b.js
b.js
b.js

which doesn't really exercise the impact of a.js on b.js

It would be nice if we could have another flag to indicate we want the tests to run like this instead:

a.js
b.js
a.js
b.js
a.js
b.js

I know there's the --shuffle option, but I'm not sure it does this exactly.

Comment 1

[Geoff Brown [:gbrown]]

One thing you can do currently is run the directory or manifest with --repeat:

./mach mochitest path/to/yourtests --repeat 3

You can comment out or delete manifest entries that you don't want to run.