Closed Bug 1558951 Opened 2 months ago Closed 2 months ago

Add support to wptrunner for allowed known_intermittent statuses

Categories

(Testing :: web-platform-tests, enhancement)

Version 3
enhancement
Not set

Tracking

(firefox69 fixed)

RESOLVED FIXED
mozilla69
Tracking Status
firefox69 --- fixed

People

(Reporter: nikkis, Assigned: nikkis, Mentored)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

Mozlog supports an additional field for statuses that are a known_intermittent. Similar functionality would be beneficial to web-platform-tests, rather than skipping or disabling tests that are intermittent. To do so, initial support for the known_intermittent field should be added to the following:

  • wpttest.py
  • manifestexpected.py - TestNode may need to be updated
  • manifestupdate.py - potentially add a KnownIntermittentUpdate class, similar to expected, or include support in ExpectedUpdate class for a known_intermittent field. - This can be part of Bug 1558954
  • metadata.py - the ExpectedUpdater class needs support added to multiple class methods to ensure a known_intermittent is recorded as expected.
  • testrunner.py - TestRunnerManager.test_ended() needs updating to reflect new known_intermittent field

This patch adds initial support for multiple allowed expected statuses in the
test metadata. A known_intermittent property has been add to the ExpectedManifest,
TestNode and SubtestNode objects. A known_intermittent(), similar to the expected()
has been added to the Test class. This defines an expected status as the first status
in the list if expected value is a list, simply the value of expected if it is a
string. A test has been written for wpttest.py to reflect these changes. Where mozlog
is used, known_intermittent has been included in the expected log.

Blocks: 1558954
Depends on: 1561421
Pushed by mjzffr@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/e5c6fb94a7eb
Add support to wptrunner for allowed known_intermittent statuses r=webdriver-reviewers,maja_zf
Status: NEW → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla69
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/17710 for changes under testing/web-platform/tests
Upstream PR merged
You need to log in before you can comment on or make changes to this bug.