Closed Bug 1631982 Opened 4 months ago Closed 3 months ago

Parsing a test path from a wpt log and feeding it back to MOZHARNESS_TEST_PATHS doesn't work

Categories

(Testing :: web-platform-tests, defect)

defect
Not set
normal

Tracking

(firefox77 fixed)

RESOLVED FIXED
mozilla77
Tracking Status
firefox77 --- fixed

People

(Reporter: khuey, Assigned: jgraham)

Details

Attachments

(1 file)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0

Steps to reproduce:

bholley pushed https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=298655447&repo=try&lineNumber=25929. Pernosco examined the log and determined that the failing test was "/preload/subresource-integrity.html".

We then synthesized the MOZHARNESS_TEST_PATHS value

'MOZHARNESS_TEST_PATHS': '{"web-platform-tests":["/preload/subresource-integrity.html"]}'

Actual results:

Rerunning the job with that MOZHARNESS_TEST_PATHS failed with

[task 2020-04-21T16:53:03.168Z] 16:53:02 CRITICAL - Unable to find any tests at the path(s):
[task 2020-04-21T16:53:03.168Z] 16:53:02 CRITICAL -   /builds/worker/workspace/build/tests/web-platform/../../../../../preload/subresource-integrity.html
[task 2020-04-21T16:53:03.168Z] 16:53:02 CRITICAL - Please check spelling and make sure there are tests in the specified path(s).

Expected results:

I'd like to run the test :)

I assume the issue is a prefix problem. It'd be nice if the test harness output and what MOZHARNESS_TEST_PATHS expected matched.

Flags: needinfo?(ahal)

Do you know if this was working previously? If so it might be a regression from:
https://bugzilla.mozilla.org/show_bug.cgi?id=1614643

I presume that MOZHARNESS_TEST_PATHS is trying to pass that in as a file system path, so it would need to be translated to the file under testing/web-platform/tests or testing/web-platform/mozilla/tests. James, I think you mentioned wptsync relied on MOZHARNESS_TEST_PATHS.. do you have a link to code that could help Kyle out?

Alternatively we could make MOZHARNESS_TEST_PATHS support WPT test ids in addition to paths.

Flags: needinfo?(ahal) → needinfo?(james)

I'm not certain but I believe it did work at one point.

Any movement or anything I can do here? mwoodrow hit this yesterday.

Making it work with test ids is going to be much better than trying to reverse engineer filesystem paths from wpt test ids. It actually looks like I did the latter for wptsync, but you have to have a wpt manifest to do that mapping properly; much better just to let the testharness do it.

It looks like the mozharness code is doing some path manipulation to turn a srcdir relative path into one that works in the slightly different path setup in mozharness. I suppose that's breaking for test ids. I'd suggest we want to skip that for anything that starts with a / (or anything that doesn't start with testing/web-platform perhaps although if it doesn't start with either then it should probably be rejected). Do you want to make a patch for that or would you like me to?

Flags: needinfo?(james)

(In reply to James Graham [:jgraham] from comment #4)

Do you want to make a patch for that or would you like me to?

If you don't mind, I'd appreciate it! Otherwise we can add this to gbrown's [dev-prod-2020] backlog in the whiteboard.

Assignee: nobody → james
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/7dedce4bde35
Support wpt test ids in MOZHARNESS_TEST_PATHS, r=ahal
Status: UNCONFIRMED → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla77
You need to log in before you can comment on or make changes to this bug.