1756047 - Allow creating Mach virtualenv in CI

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement)

Description

[Mitchell Hentges [:mhentges] 🦀]

There's cases where we need native Python packages installed in CI, but they're top-level Mach requirements (such as psutil).
However, for the current cases, we've been addressing them by creating a virtualenv, installing psutil into it, then invoking Mach with that virtualenv and pretending it's the "system Python environment".

Instead of this wackiness, let's solve this more straightforwardly:

• Mach already depends on psutil and defines the versions it needs/expects
• We've gone out of our way to avoid creating the Mach virtualenv - partially because by default it'd go in ~/.mozbuild, which may not be acceptable for long-living CI workers
• Why don't we "just" (😉) create the Mach virtualenv, but put it somewhere that's automatically cleared between tasks, like $WORKSPACE?

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1756047: When creating Mach venv in CI, put it in `$WORKSPACE`

I'm guessing that we haven't been building the Mach virtualenv in CI so
far because:

• The default location in ~/.mozbuild may not be cleared between jobs
• There's a performance cost to pip install-ing packages, especially
  if they're not needed
• In the past, it was tricky to have some jobs use a Mach virtualenv and
  others not.

This led to weird workarounds where a virtualenv would be created,
populated, then masqueraded as the "system Python" so that Mach would
be able to run with access to its dependencies without needing to
"create a Mach virtualenv".

This patch addresses the first point about the Mach virtualenv location:
In CI, if WORKSPACE is set, the Mach venv will be created in
$WORKSPACE/mach_virtualenv.

Depends on D140255

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f0043e07ec5e
When creating Mach venv in CI, put it in `$WORKSPACE` r=ahal

Comment 3

[Iulian Moraru]

Backed out for causing py3 failures with the message Exception: The "python-test" site is not compatible with the installed system Python packages.

Push with failures

Failure log

Backout link

[task 2022-03-09T22:36:53.620Z] pytest==4.6.6: Not installed
[task 2022-03-09T22:36:53.620Z] Creating default state directory: /builds/worker/.mozbuild
[task 2022-03-09T22:36:53.620Z] Running "pip check" to verify compatibility between the system Python and the "mach" site.
[task 2022-03-09T22:36:53.620Z] Running "pip check" to verify compatibility between the system Python and the "python-test" site.
[task 2022-03-09T22:36:53.620Z] Error running mach:
[task 2022-03-09T22:36:53.620Z] 
[task 2022-03-09T22:36:53.620Z]     ['python-test', '--subsuite', 'mach', '--run-slow']
[task 2022-03-09T22:36:53.620Z] 
[task 2022-03-09T22:36:53.620Z] The error occurred in code that was called by the mach command. This is either
[task 2022-03-09T22:36:53.620Z] a bug in the called code itself or in the way that mach is calling it.
[task 2022-03-09T22:36:53.620Z] You can invoke |./mach busted| to check if this issue is already on file. If it
[task 2022-03-09T22:36:53.620Z] isn't, please use |./mach busted file python-test| to report it. If |./mach busted| is
[task 2022-03-09T22:36:53.620Z] misbehaving, you can also inspect the dependencies of bug 1543241.
[task 2022-03-09T22:36:53.620Z] 
[task 2022-03-09T22:36:53.620Z] If filing a bug, please include the full output of mach, including this error
[task 2022-03-09T22:36:53.620Z] message.
[task 2022-03-09T22:36:53.620Z] 
[task 2022-03-09T22:36:53.620Z] The details of the failure are as follows:
[task 2022-03-09T22:36:53.620Z] 
[task 2022-03-09T22:36:53.620Z] Exception: The "python-test" site is not compatible with the installed system Python packages.
[task 2022-03-09T22:36:53.620Z] 
[task 2022-03-09T22:36:53.620Z]   File "/builds/worker/checkouts/gecko/python/mach_commands.py", line 146, in python_test
[task 2022-03-09T22:36:53.620Z]     return run_python_tests(command_context, *args, **kwargs)
[task 2022-03-09T22:36:53.620Z]   File "/builds/worker/checkouts/gecko/python/mach_commands.py", line 164, in run_python_tests
[task 2022-03-09T22:36:53.620Z]     command_context.activate_virtualenv()
[task 2022-03-09T22:36:53.620Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/base.py", line 863, in activate_virtualenv
[task 2022-03-09T22:36:53.620Z]     self.virtualenv_manager.activate()
[task 2022-03-09T22:36:53.620Z]   File "/builds/worker/checkouts/gecko/python/mach/mach/site.py", line 590, in activate
[task 2022-03-09T22:36:53.620Z]     self.ensure()
[task 2022-03-09T22:36:53.620Z]   File "/builds/worker/checkouts/gecko/python/mach/mach/site.py", line 564, in ensure
[task 2022-03-09T22:36:53.620Z]     if not self._up_to_date():
[task 2022-03-09T22:36:53.620Z]   File "/builds/worker/checkouts/gecko/python/mach/mach/site.py", line 762, in _up_to_date
[task 2022-03-09T22:36:53.620Z]     self._topsrcdir, pthfile_lines, self._site_name, self._requirements
[task 2022-03-09T22:36:53.620Z]   File "/builds/worker/checkouts/gecko/python/mach/mach/site.py", line 1095, in _assert_pip_check
[task 2022-03-09T22:36:53.620Z]     f'The "{virtualenv_name}" site is not compatible with the installed '
[taskcluster 2022-03-09 22:36:54.481Z] === Task Finished ===
[taskcluster 2022-03-09 22:36:54.481Z] Unsuccessful task run with exit code: 1 completed in 606.885 seconds

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/26da206aa129
When creating Mach venv in CI, put it in `$WORKSPACE` r=ahal

Comment 5

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/26da206aa129