1781195 - `./mach wpt <path to test> --product chrome` fails with "FileNotFoundError: No such file or directory: '<dist-dir>/_virtualenvs/common/bin/python'"

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect)

Description

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

+++ This bug was initially created as a clone of Bug #1777660 +++

Bug 1777660 is back. I got exactly same error (except the line numbers in mach_commands.py), and there is only build in <dist-dir>/_virtualenvs, i.e., no common directory.

I do ./mach wpt foo --product chrome on Linux, the I got:

FileNotFoundError: [Errno 2] No such file or directory: '<dist-dir>/_virtualenvs/common/bin/python'
  File "<src-dir>/testing/web-platform/mach_commands.py", line 532, in run_wpt
    return run_web_platform_tests(command_context, **params)
  File "<src-dir>/testing/web-platform/mach_commands.py", line 521, in run_web_platform_tests
    return wpt_runner.run(logger, **params)
  File "<src-dir>/testing/web-platform/mach_commands_base.py", line 57, in run
    kwargs = self.setup.kwargs_wptrun(kwargs)
  File "<src-dir>/testing/web-platform/mach_commands.py", line 188, in kwargs_wptrun
    self.virtualenv_manager.install_pip_requirements(
  File "<src-dir>/python/mach/mach/site.py", line 638, in install_pip_requirements
    install_result = self._virtualenv.pip_install(
  File "<src-dir>/python/mach/mach/site.py", line 846, in pip_install
    return subprocess.run(
  File "/usr/lib/python3.10/subprocess.py", line 501, in run
    with Popen(*popenargs, **kwargs) as process:
  File "<src-dir>/third_party/python/sentry_sdk/sentry_sdk/integrations/stdlib.py", line 190, in sentry_patched_popen_init
    rv = old_popen_init(self, *a, **kw)  # type: ignore
  File "/usr/lib/python3.10/subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1842, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)

Comment 1

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

If I run a mochitest, common directory is created and it works.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 3

[James Graham [:jgraham]]

ahochheiden: You might be the right person to ask about the supported way to fix this from a mach point of view. The basic problem is that mach wpt sometimes (when using a non-Firefox browser) needs to install stuff into the virtualenv. We don't want to repeat the logic of what to install in the mach command itself, because it's already implemented in the web-platform-tests code. So we need to ensure that we have access to the virtualenv being used inside the MozbuildObject which we need to get information about the source tree.

Currently we're doing something slightly hacky to pass the virtualenv name in explicitly (see https://searchfox.org/mozilla-central/source/testing/web-platform/mach_commands.py#506) but per this bug it isn't working. So it would be useful to have a more obvious/supported way to handle this use case (maybe mach/mozbuild should be setting up the correct virtualenv itself, rather than always defaulting to common?).

Comment 4

[Alex Hochheiden [:ahochheiden]]

I think your problem is that not all the commands are specifying virtualenv_name="wpt", so when the top-level command is called, the command_context is created with none for virtualenv (since one isn't specified, and which eventually translates to common in the virtualenv_manager). As far as I can tell, the top-level command arguments are the only ones taken into consideration.

In this case, the wpt command (without a virtualenv_name specified in the args) calls run_web_platform_tests() (which itself can be invoked via a command, but since it's not being invoked via a command, the arguments for the web-platform-tests command are discarded. Functionally, only virtualenv_name="wpt" is lost, because the rest are both identical between both commands.

Comment 5

[Alex Hochheiden [:ahochheiden]]

Attachment: Bug 1781195 - Add `virtualenv_name="wpt"` to all Web Platform Test commands r=jgraham

Comment 6

[Alex Hochheiden [:ahochheiden]]

I don't think anything else needs to be changed. At first I thought it was weird that activate was only called when a virtualenv_name was specified, but after thinking about it, it makes sense. If we don't specify a virtualenv_name, there's no virtualenv, and we stay in the mach environment, meaning there's nothing to activate.

If we invoke virtualenv_manager at a later point, it will create a set itself up for use of a common virtualenv (since one wasn't specified, and having a default seems reasonable). The only reason this was failing was because the common virtualenv wasn't being activated, and thus it didn't exist unless it was activated by doing a different command that activated/created it already. I don't think we want to use the common virtualenv, so activating it wasn't the correct fix.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3028c8c7847c
Add `virtualenv_name="wpt"` to all Web Platform Test commands r=jgraham

Comment 8

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/3028c8c7847c