1778083 - Ensure pytestrunner can only produce one result per subtest

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect)

Description

[James Graham [:jgraham]]

It appears to be possible for the pytest_runtest_logreport API to be called more than once for the same test, notably in the case where there's an error that causes the test to fail and also affects cleanup. In this case we end up with multiple results for the same test, which looks like a flaky test.

To ensure that we only get one result per test, we should switch the implementation to store results by subtest name, and overwrite any earlier results with the later result.

Comment 1

[James Graham [:jgraham]]

Attachment: Bug 1778083 - Enforce a single result per subtest for pytest results,

Previously, if we got a test failure and an error during teardown, we'd end up with
multiple results for the same test. This just picks the final result for the test.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/228851107e0f
Enforce a single result per subtest for pytest results, r=webdriver-reviewers,jdescottes

Comment 3

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/228851107e0f

Comment 4

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/34782 for changes under testing/web-platform/tests

Comment 5

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot