1701896 - Share a common implementation between the reftest harness and the WPT harness

Status: NEW

(Testing :: Marionette Client and Harness, task, P3)

Description

[Julian Descottes [:jdescottes]]

The internal reftest harness can be found at https://searchfox.org/mozilla-central/source/layout/tools/reftest/

The WPT reftest harness is at https://searchfox.org/mozilla-central/source/testing/marionette/reftest.js (+ related JSWindowActors).

Both implementations share similarities, and we are often porting logic and fixes from one to other (some examples: Bug 1600800, Bug 1701128, Bug 1574508).

Both also rely on JSWindowActors, so it seems like we could try to extract this to a shared helper/component/... used by both test harnesses.

Comment 1

[Julian Descottes [:jdescottes]]

Hi James,

As discussed on Element, can you share a few known differences/discrepancies between the two implementations?
Thanks!

Comment 2

[James Graham [:jgraham]]

Sure. I'll try to err on the side of covering more, so some of the differences aren't blockers to sharing parts of the implementation.

So the frontends to the reftest harness and wpt reftests are totally different. The reftest harness internally parses reftest.list files and uses those to build an internal list of tests. The wpt harness has common Python code to build the test list, and each test is run with a marionette command. This affects what kinds of conditionals tests can depend on since the reftest harness has access to runtime data.

The harnesses are also very different on the output side of the frontend. The reftest harness communicates results back by dumping JSON data to stdout that's later read by the Python code; the wpt harness returns the results of the test via marionette. The reftest harness also seems to have some legacy output that doesn't look like it integrates with structured logging properly, but I'm not sure.

Once you get past the frontends there are more similarities, but things still aren't quite the same:

• wpt and the reftest harness use different default window sizes (I think wpt has half-baked support for changing the window size, but iirc it isn't used so far)
• wpt reftests are always loaded over HTTP; the reftest harness defaults to loading over file:// and has a totally different server for tests that opt-in to HTTP.
• The paint-complete events have different names (MozReftestInvalidate vs TestRendered)
• The supported test types are not the same. In particular the wpt print reftests have no analouge in the reftest harness; the reftest harness has a different kind of print reftest that compares text rather than images. reftest harness also has "load" tests which are analogous to wpt crashtests, and "script" tests which don't have a wpt analouge.
• The reftest harness supports a number of features that directly interact with the layout engine in a way that we haven't got a cross-browser alternative for; https://searchfox.org/mozilla-central/source/layout/tools/reftest/README.txt#532-717 has a list of such attributes. These must do nothing in wpt tests (some similar features probably ought to be standardised for wpt reftests e.g. to allow testing zoomed states, but the machanism might look different).
• The reftest harness has support for only repainting invalidated rects. This isn't implemented in the wpt harness (although perhaps it could be aiui it's not as important given webrender and it's previously broken and been fixed without catching many regressions).
• wpt harness doesn't handle spell checking; given this isn't standardised it presumably ought to be disabled in shared wpt tests (but could be enabled in gecko-only tests ofc).

So whilst it's possible to share the implementation it might be a reasonable amount of work to do more than just share the actors. Maybe we should talk to the layout team about the way forward here; it would be good to get to a place where we're not updating the reftest harness for gecko changes and neglecting to fix the same issue in the wpt harness.