Closed Bug 1545143 Opened 5 months ago Closed 3 months ago

Orphan data (lines/whole files) in expectation ini files cause confusion


(Testing :: web-platform-tests, defect, P2)

Version 3


(firefox69 fixed)

Tracking Status
firefox69 --- fixed


(Reporter: ehsan, Assigned: jgraham)




(1 file)

This one in particular is what I'm filing the bug about:

I discovered this bug with my fix to bug 144795. In the first try push for that fix I got two unexpected pass errors in this test, but I expected to get three. The test above in particular did not emit an unexpected pass error, but that doesn't make sense given the nature of my fix. So I took that expected: FAIL line out as well and tested locally and saw the test passes. The try server agrees too: So it looks like that expected: FAIL line isn't having any effect?

The priority flag is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(james)

I think this is another case of "leftover lines cause confusion to people reading the file". I can't find a general but for the work to fix this, so I'm going to repurpose this one.

Flags: needinfo?(james)
Priority: -- → P2
Summary: Some `expected: FAIL` statements do not seem to be doing what you would expect them to do → Orphan data (lines/whole files) in expectation ini files cause confusion

This has different behaviour from the regular update in a couple of

  • It doesn't try to preserve existing conditionals that didn't match
    anything in the results
  • It removes subtests that are no longer present in the data set.

As such this is an appropriate mode to use when you have a try push
covering all the possible platforms, or when you are confident there
are no per-platform differences.

As part of this work, the code for updating the conditionals was
refactored to be clearer, and a bunch of tests were added to ensure
that it continues to work correctly.

Pushed by
Add --full flag for wpt-update for updating with a full set of results, r=maja_zf
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla69
Regressions: 1564917
Created web-platform-tests PR for changes under testing/web-platform/tests
Can't merge web-platform-tests PR due to failing upstream checks:
Github PR
* Taskcluster (pull_request) (
Upstream PR merged
Regressions: 1569559
Pushed by
[wpt PR 18211] - [Gecko Bug 1545143] Add --full flag for wpt-update for updating with a full set of results, a=testonly
You need to log in before you can comment on or make changes to this bug.