1263568 - Make dbaron happier about wptrunner's reftest support

Status: NEW

(Testing :: web-platform-tests, defect)

Description

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

David: I'm not sure if there are bugs for the things you mentioned in the email you sent me, but please make them block this bug.

NI-ing myself to nag you about it, since you're not accepting requests :)

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I'm not done.  Still need to look into:
 * assertion checking
 * whether it still uses a slow method of image comparison

Comment 3

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(And I'd also say more generally that a lot of work has gone into our existing reftest harness, and you're not going to magically produce something the same level of quality by starting over from scratch.)

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

OK, I think I've now filed the issues I'm aware of.

My preferred approach to fixing them, though, would be to use the existing reftest harness instead of trying to write a new one, rather than fixing all the bugs that I just filed.

As part of doing that, though, we should start switching to running reftests at the standard 600x600 (which is what the wpt reftest harness appears to do).  The best way to do that would probably to make that be configurable in our reftest harness on at least a per-directory basis, so that we can convert existing reftests as well (which we can then go through a directory at a time rather than all at once).  There's an existing bug on that that's been pretty quiet for a few years.

Comment 5

[James Graham [:jgraham]]

Our existing reftest harness isn't cross-browser compatible (we can't use it with Servo, for example), and doesn't afaik support continuing in the face of tests that e.g. crash the browser. Therefore it isn't suitable for running web-platform-tests (or CSSWG tests). As such I don't think that using the existing reftest harness is a viable solution.