1631982 - Parsing a test path from a wpt log and feeding it back to MOZHARNESS_TEST_PATHS doesn't work

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect)

Description

[Kyle Huey [:khuey]]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0

Steps to reproduce:

bholley pushed https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=298655447&repo=try&lineNumber=25929. Pernosco examined the log and determined that the failing test was "/preload/subresource-integrity.html".

We then synthesized the MOZHARNESS_TEST_PATHS value

'MOZHARNESS_TEST_PATHS': '{"web-platform-tests":["/preload/subresource-integrity.html"]}'

Actual results:

Rerunning the job with that MOZHARNESS_TEST_PATHS failed with

[task 2020-04-21T16:53:03.168Z] 16:53:02 CRITICAL - Unable to find any tests at the path(s):
[task 2020-04-21T16:53:03.168Z] 16:53:02 CRITICAL -   /builds/worker/workspace/build/tests/web-platform/../../../../../preload/subresource-integrity.html
[task 2020-04-21T16:53:03.168Z] 16:53:02 CRITICAL - Please check spelling and make sure there are tests in the specified path(s).

Expected results:

I'd like to run the test :)

I assume the issue is a prefix problem. It'd be nice if the test harness output and what MOZHARNESS_TEST_PATHS expected matched.

Comment 1

[Andrew Halberstadt [:ahal]]

Do you know if this was working previously? If so it might be a regression from:
https://bugzilla.mozilla.org/show_bug.cgi?id=1614643

I presume that MOZHARNESS_TEST_PATHS is trying to pass that in as a file system path, so it would need to be translated to the file under testing/web-platform/tests or testing/web-platform/mozilla/tests. James, I think you mentioned wptsync relied on MOZHARNESS_TEST_PATHS.. do you have a link to code that could help Kyle out?

Alternatively we could make MOZHARNESS_TEST_PATHS support WPT test ids in addition to paths.

Comment 2

[Kyle Huey [:khuey]]

I'm not certain but I believe it did work at one point.

Comment 3

[Kyle Huey [:khuey]]

Any movement or anything I can do here? mwoodrow hit this yesterday.

Comment 4

[James Graham [:jgraham]]

Making it work with test ids is going to be much better than trying to reverse engineer filesystem paths from wpt test ids. It actually looks like I did the latter for wptsync, but you have to have a wpt manifest to do that mapping properly; much better just to let the testharness do it.

It looks like the mozharness code is doing some path manipulation to turn a srcdir relative path into one that works in the slightly different path setup in mozharness. I suppose that's breaking for test ids. I'd suggest we want to skip that for anything that starts with a / (or anything that doesn't start with testing/web-platform perhaps although if it doesn't start with either then it should probably be rejected). Do you want to make a patch for that or would you like me to?

Comment 5

[Andrew Halberstadt [:ahal]]

(In reply to James Graham [:jgraham] from comment #4)

Do you want to make a patch for that or would you like me to?

If you don't mind, I'd appreciate it! Otherwise we can add this to gbrown's [dev-prod-2020] backlog in the whiteboard.

Comment 6

[James Graham [:jgraham]]

Attachment: Bug 1631982 - Support wpt test ids in MOZHARNESS_TEST_PATHS,

Comment 7

[James Graham [:jgraham]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5ce000b17967043e29c255ca4abf98c43a378e55&selectedTaskRun=fIS32SwpTkeA35qo6Y0QWA-0 suggests that this runs the right tests given both a path and an id as input.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7dedce4bde35
Support wpt test ids in MOZHARNESS_TEST_PATHS, r=ahal

Comment 9

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/7dedce4bde35