1504357 - `./mach wpt path/to/single/disabled.test --repeat-until-unexpected --no-pause-after-test` results in an instant infinite loop

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I'm going through disabling wpt tests that are failing on Fennec to get the rest of the tests running in CI (or in preparation for getting these tests up and running for future Geckoview-based browsers).

As part of this, I use `./mach wpt path/to/single/test.file` with the --repeat-until-unexpected and --no-pause-after-test flags passed in to help quickly re-run tests that only intermittently fail. 

I accidentally did this with a test that I had already disabled, and I ended up in an infinite loop of the test runner printing out things like this:

 0:29.56 INFO Repetition 1594
 0:29.57 SUITE_START: web-platform-test - running 1 tests
 0:29.57 INFO Running reftest tests
 0:29.57 ERROR Unsupported test type reftest for product fennec
 0:29.57 INFO Running wdspec tests
 0:29.57 ERROR Unsupported test type wdspec for product fennec
 0:29.57 INFO Running testharness tests
 0:29.57 TEST_START: /dom/events/Event-dispatch-on-disabled-elements.html
 0:29.57 TEST_END: SKIP
 0:29.57 INFO No testharness tests to run
 0:29.57 INFO Got 0 unexpected results
 0:29.57 SUITE_END

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 1594 checks (1594 tests)
Expected results: 0
Skipped: 1594 tests
OK




I feel like --repeat-until-unexpected should bail out once it realizes that there are zero tests being run.

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1504357 - Make sure --repeat-until-unexpected doesn't loop over zero tests r?jgraham

If zero tests are selected to run while --repeat-until-unexpected is specified, wptrunner can get into a fast infinite loop of running nothing until killed by the user.

This patch should stop that from happening by breaking the loop after the first iteration of nothing finishes.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f988af9437f5
Make sure --repeat-until-unexpected doesn't loop over zero tests r=jgraham

Comment 3

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/f988af9437f5

Comment 4

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/14303 for changes under testing/web-platform/tests

Comment 5

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Can't merge web-platform-tests PR due to failing upstream checks:
Github PR https://github.com/web-platform-tests/wpt/pull/14303
* continuous-integration/travis-ci/pr (https://travis-ci.org/web-platform-tests/wpt/builds/461543595?utm_source=github_status&utm_medium=notification)

Comment 6

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Can't merge web-platform-tests PR due to failing upstream checks:
Github PR https://github.com/web-platform-tests/wpt/pull/14303
* continuous-integration/travis-ci/pr (https://travis-ci.org/web-platform-tests/wpt/builds/461915235?utm_source=github_status&utm_medium=notification)

Comment 7

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/0f33ddfa45f6
[wpt PR 14303] - [Gecko Bug 1504357] Make sure --repeat-until-unexpected doesn't loop over zero tests, a=testonly

Comment 9

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/0f33ddfa45f6