1793665 - [wpt-sync] Sync PR 36154 - wptrunner: run test types in a deterministic order

Status: RESOLVED FIXED

(Testing :: web-platform-tests, task, P4)

Description

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Sync web-platform-tests PR 36154 into mozilla-central (this bug is closed when the sync is complete).

PR: https://github.com/web-platform-tests/wpt/pull/36154
Details from upstream follow.

Sam Sneddon <[e-mail]> wrote:

wptrunner: run test types in a deterministic order

Currently we don't run each test type in a constant order; make it so.

@patrickangle was trying to figure out flakiness on Azure Pipelines, and noticing significant differences in test ordering, and this is the cause. (Ultimately, because ordering of sets is undefined.)

@jgraham: I feel like we've talked about this before.

Comment 1

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Pushed to try https://treeherder.mozilla.org/#/jobs?repo=try&revision=1b4d3ef945d2ed166f07bc998c28bc2d9c303987

Comment 2

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

The PR was not expected to affect any tests, but the try push wasn't a success. Check the try results for infrastructure issues

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/df8f2e064dfd
[wpt PR 36154] - wptrunner: run test types in a deterministic order, a=testonly

Comment 4

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Test result changes from PR not available.

Comment 5

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/df8f2e064dfd