Closed Bug 1778083 Opened 3 months ago Closed 3 months ago

Ensure pytestrunner can only produce one result per subtest

Categories

(Testing :: web-platform-tests, defect)

Default
defect

Tracking

(firefox104 fixed)

RESOLVED FIXED
104 Branch
Tracking Status
firefox104 --- fixed

People

(Reporter: jgraham, Assigned: jgraham)

Details

Attachments

(1 file)

It appears to be possible for the pytest_runtest_logreport API to be called more than once for the same test, notably in the case where there's an error that causes the test to fail and also affects cleanup. In this case we end up with multiple results for the same test, which looks like a flaky test.

To ensure that we only get one result per test, we should switch the implementation to store results by subtest name, and overwrite any earlier results with the later result.

Previously, if we got a test failure and an error during teardown, we'd end up with
multiple results for the same test. This just picks the final result for the test.

Assignee: nobody → james
Status: NEW → ASSIGNED
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/228851107e0f
Enforce a single result per subtest for pytest results, r=webdriver-reviewers,jdescottes
Status: ASSIGNED → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → 104 Branch
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/34782 for changes under testing/web-platform/tests
Upstream PR merged by moz-wptsync-bot
You need to log in before you can comment on or make changes to this bug.