1558951 - Add support to wptrunner for allowed known_intermittent statuses

Status: RESOLVED FIXED

(Testing :: web-platform-tests, enhancement)

Description

[Nikki Sharpley [:nikkis]]

Mozlog supports an additional field for statuses that are a known_intermittent. Similar functionality would be beneficial to web-platform-tests, rather than skipping or disabling tests that are intermittent. To do so, initial support for the known_intermittent field should be added to the following:

• wpttest.py
• manifestexpected.py - TestNode may need to be updated
• manifestupdate.py - potentially add a KnownIntermittentUpdate class, similar to expected, or include support in ExpectedUpdate class for a known_intermittent field. - This can be part of Bug 1558954
• metadata.py - the ExpectedUpdater class needs support added to multiple class methods to ensure a known_intermittent is recorded as expected.
• testrunner.py - TestRunnerManager.test_ended() needs updating to reflect new known_intermittent field

Comment 1

[Nikki Sharpley [:nikkis]]

Attachment: Bug 1558951 - Add support to wptrunner for allowed known_intermittent statuses

This patch adds initial support for multiple allowed expected statuses in the
test metadata. A known_intermittent property has been add to the ExpectedManifest,
TestNode and SubtestNode objects. A known_intermittent(), similar to the expected()
has been added to the Test class. This defines an expected status as the first status
in the list if expected value is a list, simply the value of expected if it is a
string. A test has been written for wpttest.py to reflect these changes. Where mozlog
is used, known_intermittent has been included in the expected log.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e5c6fb94a7eb
Add support to wptrunner for allowed known_intermittent statuses r=webdriver-reviewers,maja_zf

Comment 3

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/e5c6fb94a7eb

Comment 4

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/17710 for changes under testing/web-platform/tests

Comment 5

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged