1528735 - Wptrunner quits early when repeating a set of tests including a skipped test

Status: RESOLVED FIXED

(Testing :: web-platform-tests, enhancement)

Description

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Bug 1504357 added a if len(test_loader.test_ids) == skipped_tests: block at https://searchfox.org/mozilla-central/rev/ee40541496d3ad738097eebadaf4965ca1343b7a/testing/web-platform/tests/tools/wptrunner/wptrunner/wptrunner.py#292.

Unfortunately the skipped_tests counter is not reset when we pass a "repeat" argument. For example, when running three tests, of which one is skipped, five times, the condition will be true on the third iteration and the fourth and fifth iterations will be lost. This causes issues with the stability checker like here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c65e29d5642a966c7cb8e8074ef4b6a90eb8a884&selectedJob=228993605.

Comment 1

[James Graham [:jgraham]]

Attachment: Bug 1528735 - Only stop if all tests are skipped on the first iteration,

If we got past the first iteration we shouldn't stop for the skipped count matching
the total tests since we increment this with the total tests on each iteration.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6d5a1544d885
Only stop if all tests are skipped on the first iteration, r=Ms2ger

Comment 3

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/15443 for changes under testing/web-platform/tests

Comment 4

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream web-platform-tests status checks passed, PR will merge once commit reaches central.

Comment 5

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/6d5a1544d885

Comment 6

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged