916797 - Mochitest(-plain/browser/chrome) should support run-until-failure when running multiple tests

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[:Gijs (he/him)]

... so that when used with --keep-open, it just runs tests until the test you're interested in fails. Right now debugging test failures that depend on the mess that earlier tests made of the browser window is a pain, and you can't use --keep-open in a useful way (also because there's no way to specify multiple file paths). We should fix this.

Comment 1

[:Felipe Gomes (needinfo for replies!)]

Attachment: 916797

Run-until-failure as implemented (bug 875463) right now works by stopping execution at the "reloop" step of a test run, which made it mostly useful for single tests run.

But there are more use cases now where run-until-failure is useful to be used with more than 1 test running, so this patch makes it able to stop execution at the "next test" step, as it's meant to be.

For m-plain (TestRunner.js) it was easy, as there's an existing _haltTests functionality that I can use (and fix reloop to respect that flag which it didn't before)

For m-chrome|browser (browser-test.js) I tried just calling TestRunner.finish, which kinda worked but when it was called there were already scheduled clean-up functions that would print errors because there were being called at the wrong time. So I implemented an analogous haltTests function that just skips whatever tests are remaining and let the harness do its clean-up properly.

Comment 2

[:Gijs (he/him)]

Comment on attachment 826995 [details] [diff] [review]
916797

Review of attachment 826995 [details] [diff] [review]:
-----------------------------------------------------------------

From a casual glance, this looks OK. Thanks for working on this!

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 826995 [details] [diff] [review]
916797

Review of attachment 826995 [details] [diff] [review]:
-----------------------------------------------------------------

this looks very useful, thanks!

Comment 4

[:Felipe Gomes (needinfo for replies!)]

https://hg.mozilla.org/integration/fx-team/rev/34307fd1289e

Comment 5

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/34307fd1289e