1294820 - Support running web-platform-tests from interactive loaner

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect)

Description

[Andrew Halberstadt [:ahal]]

We should add a test package command for running web-platform-tests.

Taking a quick look at testing/web-platform/mach_commands.py, this might be a bit more involved than the other harnesses because there is a lot of MozbuildObject dependent logic in the mach command (with other harnesses, the mach_commands.py was refactored into the harness, and relatively empty).

My general approach has been to create a new mach_test_package_commands.py file to keep the two mach environments separate. However there is still a lot of code that should be shared between these two mach commands. Ideally this would be refactored into the test harness, but I'm not sure if that's possible given wpt's upstream repo.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

getting a 1-click loaner and using option #2, yields a prompt with stuff downloaded, but unlike mochitests, there is no venv setup and activated.

Comment 4

[Andrew Halberstadt [:ahal]]

WPT has been modified to run from the srcdir instead of the test package. So the steps to get this supported will differ slightly from other harnesses. I think ideally we modify the existing mach command to not depend on an objdir, and just re-use that from the loaner.

Comment 5

[Andrew Halberstadt [:ahal]]

More context:
<ahal> jgraham: in case you're very anxious to get this moving, the next step will be to detect if there is a srcdir and add the 'mach' binary to $PATH there instead of this: https://dxr.mozilla.org/mozilla-central/source/taskcluster/scripts/tester/run-wizard#53
<ahal> and then after that, it is just a matter of modifying the mach command to provide intelligent defaults (i.e automatically find the firefox binary that mozharness downloaded, etc)

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

a recent attempt at this:
elect one of the above options: 2
Timed out after 30 seconds waiting for the 'run-mozharness' binary
run-mozharness --no-run-tests
Traceback (most recent call last):
  File "./run-wizard", line 170, in <module>
    sys.exit(wizard())
  File "./run-wizard", line 163, in wizard
    ret = func()
  File "./run-wizard", line 47, in setup
    status = call(['run-mozharness', '--no-run-tests'])
  File "./run-wizard", line 20, in call
    return subprocess.call(cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 523, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
root@taskcluster-worker:~# 

not sure if we can get to this sometime in the near future.

Comment 7

[James Graham [:jgraham]]

Attachment: Bug 1294820 - Add support for running wpt from one-click loaner mach environment,

Allow running |mach wpt| on one click loaners in order to run
web-platform-tests tests.

This implementation is just like the one for other testsuites using
thee packaged tests rather than the checkout that we get with wpt, at
least on Linux. That's also where the tests run from so it seems
reasonable for now. Moving to the checkout in the future could remove
some of the logic here by using a fake mozbuild environment so that
the testsuite itself doesn't have to implement anything much.

Review commit: https://reviewboard.mozilla.org/r/151012/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/151012/

Comment 8

[Andrew Halberstadt [:ahal]]

Comment on attachment 8879645 [details]
Bug 1294820 - Add support for running wpt from one-click loaner mach environment,

https://reviewboard.mozilla.org/r/151012/#review155800

Thanks for tackling this! Should we handle the wizard issue in a follow-up?

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/21c963bcc2ce
Add support for running wpt from one-click loaner mach environment, r=ahal

Comment 10

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/21c963bcc2ce