Closed Bug 1545143 Opened 5 months ago Closed 3 months ago

Orphan data (lines/whole files) in expectation ini files cause confusion

Categories

(Testing :: web-platform-tests, defect, P2)

Version 3
defect

Tracking

(firefox69 fixed)

RESOLVED FIXED
mozilla69
Tracking Status
firefox69 --- fixed

People

(Reporter: ehsan, Assigned: jgraham)

References

Details

Attachments

(1 file)

This one in particular is what I'm filing the bug about:
https://searchfox.org/mozilla-central/rev/d33d470140ce3f9426af523eaa8ecfa83476c806/testing/web-platform/meta/html/dom/interfaces.https.html.ini#207

I discovered this bug with my fix to bug 144795. In the first try push for that fix https://treeherder.mozilla.org/#/jobs?repo=try&revision=ef0eb8dd51631a4da3f51b3f0115b97906ba5a57 I got two unexpected pass errors in this test, but I expected to get three. The test above in particular did not emit an unexpected pass error, but that doesn't make sense given the nature of my fix. So I took that expected: FAIL line out as well and tested locally and saw the test passes. The try server agrees too: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9cbd2e28896ccdd90964cac56fda89ae5bdccc3e. So it looks like that expected: FAIL line isn't having any effect?

The priority flag is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(james)

I think this is another case of "leftover lines cause confusion to people reading the file". I can't find a general but for the work to fix this, so I'm going to repurpose this one.

Flags: needinfo?(james)
Priority: -- → P2
Summary: Some `expected: FAIL` statements do not seem to be doing what you would expect them to do → Orphan data (lines/whole files) in expectation ini files cause confusion

This has different behaviour from the regular update in a couple of
ways:

  • It doesn't try to preserve existing conditionals that didn't match
    anything in the results
  • It removes subtests that are no longer present in the data set.

As such this is an appropriate mode to use when you have a try push
covering all the possible platforms, or when you are confident there
are no per-platform differences.

As part of this work, the code for updating the conditionals was
refactored to be clearer, and a bunch of tests were added to ensure
that it continues to work correctly.

Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/3bce3ca779f0
Add --full flag for wpt-update for updating with a full set of results, r=maja_zf
Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla69
Regressions: 1564917
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/18211 for changes under testing/web-platform/tests
Can't merge web-platform-tests PR due to failing upstream checks:
Github PR https://github.com/web-platform-tests/wpt/pull/18211
* Taskcluster (pull_request) (https://tools.taskcluster.net/task-group-inspector/#/HUaqV_mcRyabdTQ5qUexPg)
Upstream PR merged
Regressions: 1569559
Pushed by wptsync@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ba28105483cd
[wpt PR 18211] - [Gecko Bug 1545143] Add --full flag for wpt-update for updating with a full set of results, a=testonly
You need to log in before you can comment on or make changes to this bug.