1598163 - If a WPT reftest has both "match" and "mismatch" references, the harness only tests the "match" one and ignores the "mismatch" one

Status: RESOLVED INVALID

(Testing :: web-platform-tests, defect)

Description

[Daniel Holbert [:dholbert]]

STR:
(1) Apply https://phabricator.services.mozilla.com/D44583 if it hasn't already landed (I'll be landing it shortly, at which point this step won't be necessary). This patch adds a testcase "percentage-height-quirk-excludes-flex-grid-002.html" that has both a "match" and a "mismatch" reference case, each specified via link tags.

(2) Delete the first line of testing/web-platform/tests/quirks/reference/percentage-height-quirk-excludes-flex-grid-002-notref.html (which is <DOCTYPE html>) -- after that change, this "notref" file will render identically to the testcase.

(3) Run the WPT test:

./mach wpt testing/web-platform/tests/quirks/percentage-height-quirk-excludes-flex-grid-002.html

EXPECTED RESULTS:
The WPT harness should complain, saying that the percentage-height-quirk-excludes-flex-grid-002.html != percentage-height-quirk-excludes-flex-grid-002-notref.html comparison unexpectedly failed (i.e. the files ended up rendering identically, which is unexpected according to the link tag)

ACTUAL RESULTS:
The WPT run completes successfully, with no mention of the -notref file.

Note: If I delete the <link rel="match" ...> file in the testcase (leaving the <link rel="mismatch" tag behind as the only reference-case line), then I get expected results. So it seems like the WPT harness is only letting me specify a single reference case here, and in particular, ignoring the "mismatch" line when there's a "match" line present.

This disagrees with the documentation at https://web-platform-tests.org/writing-tests/reftests.html , so I'm guessing it's a harness bug.

Comment 1

[Daniel Holbert [:dholbert]]

Ah, I misread the documentation slightly -- on a reread, it looks like if you specify multiple link tags in the same testcase, then the test passes if any one link tag is satisfied. (This matches my observed behavior in my STR, so maybe my "expected results" are incorrect, and my "ACTUAL RESULTS" are expected after all.)

It looks like the documentation says that you want all of the link tags to be honored (multiple comparisons for a single test), then you have to chain them by putting the of the link tag for the second comparison in the reference case. But I tried doing this (as an alternative approach, earlier), and it still didn't work... perhaps because the reference case lives in a subdirectory and so the test harness isn't finding and processing it's link tag, perhaps? I'll come up with some STR to demonstrate...

I'm also not sure how this setup would allow you to specify that a single testcase has multiple "mismatch" reference cases... e.g. theoretically, is it possible to represent this scenario below?

!= test-1.html test-1-notref-a.html
!= test-1.html test-1-notref-b.html

I don't think the WPT link tag rules allow for that... If you list both mismatch link tags in the same file, then the test would pass as soon as we've checked the first one (assuming it mismatches), and we'd never check the second. And there's no way to "chain" them usefully, either.

Comment 2

[Daniel Holbert [:dholbert]]

(In reply to Daniel Holbert [:dholbert] from comment #1)

It looks like the documentation says that you want all of the link tags to be honored (multiple comparisons for a single test), then you have to chain them by putting the of the link tag for the second comparison in the reference case. But I tried doing this (as an alternative approach, earlier), and it still didn't work... perhaps because the reference case lives in a subdirectory and so the test harness isn't finding and processing it's link tag, perhaps? I'll come up with some STR to demonstrate...

Ah, my bad -- I ended up with the wrong path in the "href" attribute, when I cut and pasted the <link rel="mismatch"> tag from my testcase to my reference case. When I fixed that, it passed/failed in the way that I expect it should. So all is well here. So we match the documented behavior after all.

I'll go ahead and close this as invalid (sorry for the noise).

Though: I am still curious how we'd represent a "single testcase with no 'match' reference cases but with several 'mismatch' reference cases" hypothetical scenario that I brought up in comment 1...

Comment 3

[James Graham [:jgraham]]

So, well done you stepped into a hornet nest :)

Basically the model that wpt adopted was different from the model CSSWG thought that they'd agreed to, and no one noticed for several years. After some recent discussion there's a plan to change the rules to something that seems to match author expectations better and is closer to what you were looking for in comment #0.

https://github.com/web-platform-tests/rfcs/pull/15 is the RFC; I think gsnedders is going to implement the change at some point.

Comment 4

[Sam Sneddon [:gsnedders]]

Basically the model that wpt adopted was different from the model CSSWG thought that they'd agreed to, and no one noticed for several years.

Well, the model WPT adopted was intended to be the same as what the CSS WG used, but somehow the WPT model diverged from the CSS WG documentation of their model, and the CSS WG agreed to it given the belief it was the same as what they already used.

I hope to implement the change soon, preferably in the remainder of this month.

Comment 5

[Daniel Holbert [:dholbert]]

That's great to hear - thanks!