Closed Bug 1558954 Opened 5 years ago Closed 5 years ago

Update the expected test metadata format to include a known_intermittent option.

Categories

(Testing :: web-platform-tests, enhancement)

Version 3
enhancement
Not set
normal

Tracking

(firefox70 fixed)

RESOLVED FIXED
mozilla70
Tracking Status
firefox70 --- fixed

People

(Reporter: nikkis, Assigned: nikkis, Mentored)

References

Details

Attachments

(1 file, 1 obsolete file)

The metadata for tests are stored in ini-like files with a very specific format. To keep up with the known_intermittent support, the way in which the metadata is updated will need amending. As the known_intermittent property is a list, this will need to be taken into account when comparing expected data with actual results. Changes to manifestupdate.py are required.

Summary: Update the expected test metadata format to include known_intermittent option. → Update the expected test metadata format to include a known_intermittent option.
Depends on: 1558951
Depends on: 1561421

Currently, the expected field in the test metadata accepts only one status. This patch adds
the ability to include known intermittent statuses in this metadata.

The existing metadata format is:

[test]
  [subtest]
    expected: PASS

This new format, if there is a known intermittent status to record, will be:

[test]
  [subtest]
    expected: [PASS, FAIL]

The first status will always be the "expected" status. The following statuses in the list are
"known intermittent" statuses. The statuses are ordered based on tree.result_values counts
during build_tree().

Tests have been added to test_update.py to account for the following circumstances:

  • an existing metadata file with only one expecte status needs updating with a known intermittent
  • a file with an existing known intermittent needs to be updated with an additional status
  • an intermittent status is now the expected status, and the expected status is now intermittent
  • a new metadata file needs to be created with a known intermittent status (WORK IN PROGRESS)

testrunner.py has been updated to exclude tests with a known_intermittent status from the
unexpected_count.

A test has been added to test_expectedtree.py to ensure the tree.result_values is counting
correctly. (WORK IN PROGRESS)

Work in progress - manifestupdate.py and metadata.py still require changes that snowball from
expectedtree.py changes. The actual update functionality has not been changed yet.

Assignee: nobody → nsharpley
Blocks: 1567000

Changed disable_intermittent back to stability at command line. This follows on from diff
D37729.

Attachment #9079110 - Attachment is obsolete: true
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/fbc5881fd4b6
Update the expected test metadata format to incude known intermittent statuses r=jgraham
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla70
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/18230 for changes under testing/web-platform/tests
Pushed by wptsync@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f67a8edc699e
[wpt PR 18230] - [Gecko Bug 1558954] Update the expected test metadata format to incude known intermittent statuses, a=testonly
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: