1533784 - Need a way to create wpt jobs that are subsetted to specific test directories, without conflicting with MOZ_HARNESS_PATHS (specified by try fuzzy, for example)

Status: NEW

(Testing :: web-platform-tests, enhancement)

Description

[Byron Campen [:bwc]]

In bug 1531489, I need to create new CI jobs that run just the webrtc wpt with some prefs set (this is the "spi" variant that exists in mochitest already). One way of doing this is by specifying MOZ_HARNESS_PATHS in the job description (in https://searchfox.org/mozilla-central/source/taskcluster/ci/test/web-platform.yml). However, it seems like this might conflict with MOZ_HARNESS_PATHS set by mach try fuzzy. For example:

"./mach try fuzzy <some non-webrtc wpt path>" ought not to run the new spi jobs at all

"./mach try fuzzy <some non-webrtc wpt path> testing/web-platform/tests/webrtc" ought to run spi for only the webrtc tests, and ordinary e10s for both the webrtc and non-webrtc test paths

"./mach try fuzzy testing/web-platform/tests/webrtc/testing/web-platform/tests/webrtc/RTCRtpTransceiver.https.html" ought to run that specific test in both spi and ordinary e10s mode

We probably need something similar to the tags we use in mochitest.

Comment 1

[Andrew Halberstadt [:ahal]]

Yeah, I don't think we should be using MOZHARNESS_TEST_PATHS. That is pretty hacky, suitable for use on try maybe, but I wouldn't want to start building actual tasks on top of it. I think we'll be shifting more towards path based scheduling in the future and will likely deprecated MOZHARNESS_TEST_PATHS when that happens.

In the meantime, I think tags are what we want. Just looking at |mach web-platform-tests --help| it looks like tags might already be supported? Note, we'd need to add a --tag argument to the mozharness script to support this:
https://searchfox.org/mozilla-central/source/testing/mozharness/scripts/web_platform_tests.py#36

Alternatively I guess we could make a full wpt flavour for this, but that seems like it might be overkill.

James, do you know if there's any precedent in wpt for something like this?

Comment 2

[James Graham [:jgraham]]

There isn't any precedent; I suggested using MOZHARNESS_TEST_PATHS because it seems like the only supported way at the moment to pass in a list of paths. Why do we think it's a bad idea?

wpt does support tags, theoretically, but afaik no one ever tried to use them and I sort of regard the entire thing as a failed experiment. It's also a little tricky with wpt since the set of tests isn't fixed so you really want to specify a directory rather than a file. I think the tagging thing has some auto-creation of per-directory tags, but really it seems better to just pass in the directories directly rather than via a layer of tagging indirection.

So I still think that passing paths is the right thing to do, but if we don't want to use the environment variable to do it then the other option is to add another argument to the web_platform_tests.py mozharness script that accepts paths that get passed to the test running command. And then the task definition can just add that to the task-specific arguments it passes in.

Comment 3

[Andrew Halberstadt [:ahal]]

The problem with MOZHARNESS_TEST_PATHS is that when passing in paths with |mach try| it's going to get overwritten. So I think we'll need to add the test path argument to the mozharness script and update the logic around what happens when MOZHARNESS_TEST_PATHS also exists (e.g, take the intersection of paths I guess?).

Also:

"./mach try fuzzy <some non-webrtc wpt path>" ought not to run the new spi jobs at all

"./mach try fuzzy <some non-webrtc wpt path> testing/web-platform/tests/webrtc" ought to run spi for only the webrtc tests, and ordinary e10s for both the webrtc and non-webrtc test paths

"./mach try fuzzy testing/web-platform/tests/webrtc/testing/web-platform/tests/webrtc/RTCRtpTransceiver.https.html" ought to run that specific test in both spi and ordinary e10s mode

I anticipate this being a bit difficult. The information used to filter tasks down by path all comes out of the build system and depends on fairly heavyweight definitions of a "suite". At a minimum, we'd need to add an entry to the list in testing/mozbase/moztest/moztest/resolve.py, but probably also some logic in there to filter by test paths as well.

Comment 4

[Alastor Wu [:alwu]]

I also have a similar need to create a new CI jobs based on the specific test directory. On Android emulator, we got a issue (bug 1546128) which affects almost all of our WebVTT rendering wpts.

Therefore, I has been thinking about if it's possible to bunch these tests and create a new CI job for them, and runs it on Android devices, not on emulator.