Update the expected test metadata format to include a known_intermittent option.
Categories
(Testing :: web-platform-tests, enhancement)
Tracking
(firefox70 fixed)
Tracking | Status | |
---|---|---|
firefox70 | --- | fixed |
People
(Reporter: nikkis, Assigned: nikkis, Mentored)
References
Details
Attachments
(1 file, 1 obsolete file)
The metadata for tests are stored in ini-like files with a very specific format. To keep up with the known_intermittent support, the way in which the metadata is updated will need amending. As the known_intermittent
property is a list, this will need to be taken into account when comparing expected data with actual results. Changes to manifestupdate.py are required.
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Comment 1•5 years ago
|
||
Currently, the expected
field in the test metadata accepts only one status. This patch adds
the ability to include known intermittent statuses in this metadata.
The existing metadata format is:
[test]
[subtest]
expected: PASS
This new format, if there is a known intermittent status to record, will be:
[test]
[subtest]
expected: [PASS, FAIL]
The first status will always be the "expected" status. The following statuses in the list are
"known intermittent" statuses. The statuses are ordered based on tree.result_values
counts
during build_tree()
.
Tests have been added to test_update.py to account for the following circumstances:
- an existing metadata file with only one expecte status needs updating with a known intermittent
- a file with an existing known intermittent needs to be updated with an additional status
- an intermittent status is now the expected status, and the expected status is now intermittent
- a new metadata file needs to be created with a known intermittent status (WORK IN PROGRESS)
testrunner.py has been updated to exclude tests with a known_intermittent
status from the
unexpected_count
.
A test has been added to test_expectedtree.py to ensure the tree.result_values
is counting
correctly. (WORK IN PROGRESS)
Work in progress - manifestupdate.py and metadata.py still require changes that snowball from
expectedtree.py changes. The actual update functionality has not been changed yet.
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Comment 2•5 years ago
|
||
Changed disable_intermittent
back to stability
at command line. This follows on from diff
D37729.
Updated•5 years ago
|
Pushed by james@hoppipolla.co.uk: https://hg.mozilla.org/integration/autoland/rev/fbc5881fd4b6 Update the expected test metadata format to incude known intermittent statuses r=jgraham
Comment 4•5 years ago
|
||
bugherder |
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/18230 for changes under testing/web-platform/tests
Can't merge web-platform-tests PR due to failing upstream checks: Github PR https://github.com/web-platform-tests/wpt/pull/18230 * Taskcluster (pull_request) (https://tools.taskcluster.net/task-group-inspector/#/DdV2b0_jSmO6Rr5NUUAwhQ)
Can't merge web-platform-tests PR due to failing upstream checks: Github PR https://github.com/web-platform-tests/wpt/pull/18230 * Taskcluster (pull_request) (https://tools.taskcluster.net/task-group-inspector/#/ajNqlf7RQyiBlDQIfXk-yQ)
Upstream PR merged
Pushed by wptsync@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/f67a8edc699e [wpt PR 18230] - [Gecko Bug 1558954] Update the expected test metadata format to incude known intermittent statuses, a=testonly
Comment 10•5 years ago
|
||
bugherder |
Description
•