1755516 - Only use system python packages if requested by environment variable

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement, P2)

Description

[Mitchell Hentges [:mhentges] 🦀]

Earlier discussion:

• 2021 Sept - thoughts for handling CI
• 2021 Dec - consideration for portage

---

Our Firefox builds must execute without having to lean on pypi.org. However, some functionality associated with our builds require unvendorable (native) Python packages (such as zstandard, which has native, platform-specific code).

This is solved by installing necessary packages to the host system's Python environment, then having Mach automatically use such packages.

However, the current implementation for how this is handled is:

1. If MOZ_AUTOMATION or MACH_USE_SYSTEM_PYTHON isn't set, then Mach does "normal" operation: a Mach virtualenv is created with native packages installed via pip, then each command virtualenv is populated with other packages with pip as needed. The system python's packages are never used, and this case is not relevant to this bug. Otherwise, if either variable is set:
2. Check if the system environment has pip. If not, then operate Mach using only vendored packages
3. Using pip, check if the system environment provides any native packages. If not, then operate Mach using only vendored packages.
4. Verify that the system python's packages are compatible with Mach's vendored packages and version requirements. If not, then bail out with an error
5. Run Mach, importing packages from the system as possible.

This resolved most use cases for us:

• The "use native packages without communicating with pypi.org" case
• The developer use case where pip install is OK
• Older python environments generally didn't provide any native packages that Mach would optionally use, so it would conveniently slide into the "ignore system python" case
• Python environments without pip would also conveniently be ignored

However, here's the use cases that were handled poorly:

• There's cases in CI where a system python package is needed, but pip isn't installed.
• There's cases in CI where we're not doing a build, so communicating with pypi.org is acceptable. However, we're still jumping through the "is system python?" hoops because MOZ_AUTOMATION is set when Mach is initializing and hasn't parsed the specific Mach command yet.
• Users of Gentoo want to build Firefox without having ~/mozbuild/srcdirs/*/_virtualenvs/mach populated, but have a myriad of different Python configurations: some of which were triggering Mach's "use system python" path, often running into "package compatibility check" failures. These users almost always don't need the features provided by Mach having access to native packages, and should instead always build with only vendored packages.

---

The solution here is to allow the use of a MACH_NATIVE_PACKAGE_SOURCE environment variable:

• If unset, or set to pip, then do pip install from pypi.org
• If set to system, then assert that the system packages are compatible (erroring if it isn't) and using them accordingly
• If set to none, then only operate with vendored packages
• Finally, if MACH_NATIVE_PACKAGE_SOURCE=system is run with any Mach command other than build, then fail with an error
  • This will allow us to drop a lot of compatibility requirements for command virtualenv packages, since they'll only have to be compatible with the top-level Mach virtualenv lockfile (which isn't applied when consuming from the system).

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

As part of this work, let's have MACH_NATIVE_PACKAGE_SOURCE replace MACH_USE_SYSTEM_PYTHON.
Ideally, we could've done this work all while using MACH_USE_SYSTEM_PYTHON instead, but it's hard to represent the "don't use either the system or pip install" case when the variable is specifically called "use system python".

Maintaining both is not ideal for both decision-making complexity and confusion for consumers.
The breaking change of removing it should be relatively low impact since:

• MACH_USE_SYSTEM_PYTHON hasn't existed for that long (2-ish years?)
• It has a direct replacement: MACH_NATIVE_PACKAGE_SOURCE=system.

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1755516: Expose config variable for Mach native dependency source

Adds support and documentation for the
MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE environment variable.

The "fallback" behaviour here should be backwards-compatible, which will
enable us to piece-by-piece explicitly use the "system" where
needed, then remove the backwards-compatibility shim afterwards.

A restriction added by this change is that it's no longer possible to
have the combination of:

• Have Mach use the system Python, and
• Have the Mach command site do pip install.

This is because this case shouldn't be necessary (if pip install is
allowed, why use the system?) and will allow us to enforce that all
non-build command sites always use their lockfiles, when they're set up.

When referring to behaviour in CI, I've opted to only refer to the
upcoming behaviour (of MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none")
because the current heuristic behaviour feels complex enough that it
should be understood by checking the code, rather than docs. Besides,
the heuristic will be removed pretty soon (right? ;)

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1755516: Add "common" to PIP_NETWORK_INSTALL_RESTRICTED_VIRTUALENVS

There are cases in CI where we're using the system Python, but we
still want to create a virtualenv. For example, build jobs use the
system Python packages, but Mozharness wants to have access to Mach
packages. So, a virtualenv needs to be created (the "common"
virtualenv), whose associated python binary is used to invoke
Mozharness.

So, just like for the build site, allow creating the common
site even when the "native package source" is the system Python.

Since mach_virtualenv_requirements.txt already depends on
common_virtualenv_requirements.txt, this restriction should not cause
validation breakage.

Depends on D138932

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1755516: Explicitly use system Python packages for Firefox build CI

Updates all build-related jobs (artifact-build, build and
instrumented-build) tasks to explicitly set
MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE=system. This allows them
to consume psutil (if installed on the system) without needing
to hit PyPI.org.

Modifies build-l10n.sh and build-linux.sh to no longer
explicitly fetch psutil from PyPI
(./mach python --virtualenv psutil), since that is replaced
by Mach's "native package source" behaviour.

Depends on D140256

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1755516: Remove "use system Python packages" inference

Simplify Mach's decision-making in CI for where to find native Python
packages: unless MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE=system is
provided, always ignore system packages.

Removes optional dependency on system's pip, instead leaning on
vendored pip copy to see what package versions are installed.

Depends on D140257

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1755516: Simplify "site.py" handling of command site package source

Now that we are gleaning a site packages source in a process-global way
(that is not conditional on the site that's being managed), we've
removed an edge case from occurring: the case where Mach would use
SitePackagesSource.NONE, but the command site would use
SitePackagesSource.SYSTEM. This would've occurred if none of Mach's
dependencies were found in the system Python, but instead once the
command site was initialized, one of its dependencies were located.

Since that can no longer happen:

• Command sites that don't populate their VENV will always use the
  same site packages source as Mach
• pthfile generation is simplified (the priority of the system paths
  no longer variable)
• We no longer need to track site_packages_source in metadata, since
  we can use mach_site_packages_source instead.

While here, I moved the "PIP_NETWORK_INSTALL_RESTRICTED_VIRTUALENVS
sites shouldn't have pypi_requirements" check to a test, since there's
no need for it to happen at runtime.

Depends on D140258

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5f956232e850
Expose config variable for Mach native dependency source r=ahal
https://hg.mozilla.org/integration/autoland/rev/9fc187cb982e
Add "common" to PIP_NETWORK_INSTALL_RESTRICTED_VIRTUALENVS r=ahal

Comment 9

[Iulian Moraru]

Backed out for causing py3 failures with the message Exception: The "python-test" site is not compatible with the installed system Python packages.

Push with failures

Failure log

Backout link

[task 2022-03-09T22:36:53.620Z] pytest==4.6.6: Not installed
[task 2022-03-09T22:36:53.620Z] Creating default state directory: /builds/worker/.mozbuild
[task 2022-03-09T22:36:53.620Z] Running "pip check" to verify compatibility between the system Python and the "mach" site.
[task 2022-03-09T22:36:53.620Z] Running "pip check" to verify compatibility between the system Python and the "python-test" site.
[task 2022-03-09T22:36:53.620Z] Error running mach:
[task 2022-03-09T22:36:53.620Z] 
[task 2022-03-09T22:36:53.620Z]     ['python-test', '--subsuite', 'mach', '--run-slow']
[task 2022-03-09T22:36:53.620Z] 
[task 2022-03-09T22:36:53.620Z] The error occurred in code that was called by the mach command. This is either
[task 2022-03-09T22:36:53.620Z] a bug in the called code itself or in the way that mach is calling it.
[task 2022-03-09T22:36:53.620Z] You can invoke |./mach busted| to check if this issue is already on file. If it
[task 2022-03-09T22:36:53.620Z] isn't, please use |./mach busted file python-test| to report it. If |./mach busted| is
[task 2022-03-09T22:36:53.620Z] misbehaving, you can also inspect the dependencies of bug 1543241.
[task 2022-03-09T22:36:53.620Z] 
[task 2022-03-09T22:36:53.620Z] If filing a bug, please include the full output of mach, including this error
[task 2022-03-09T22:36:53.620Z] message.
[task 2022-03-09T22:36:53.620Z] 
[task 2022-03-09T22:36:53.620Z] The details of the failure are as follows:
[task 2022-03-09T22:36:53.620Z] 
[task 2022-03-09T22:36:53.620Z] Exception: The "python-test" site is not compatible with the installed system Python packages.
[task 2022-03-09T22:36:53.620Z] 
[task 2022-03-09T22:36:53.620Z]   File "/builds/worker/checkouts/gecko/python/mach_commands.py", line 146, in python_test
[task 2022-03-09T22:36:53.620Z]     return run_python_tests(command_context, *args, **kwargs)
[task 2022-03-09T22:36:53.620Z]   File "/builds/worker/checkouts/gecko/python/mach_commands.py", line 164, in run_python_tests
[task 2022-03-09T22:36:53.620Z]     command_context.activate_virtualenv()
[task 2022-03-09T22:36:53.620Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/base.py", line 863, in activate_virtualenv
[task 2022-03-09T22:36:53.620Z]     self.virtualenv_manager.activate()
[task 2022-03-09T22:36:53.620Z]   File "/builds/worker/checkouts/gecko/python/mach/mach/site.py", line 590, in activate
[task 2022-03-09T22:36:53.620Z]     self.ensure()
[task 2022-03-09T22:36:53.620Z]   File "/builds/worker/checkouts/gecko/python/mach/mach/site.py", line 564, in ensure
[task 2022-03-09T22:36:53.620Z]     if not self._up_to_date():
[task 2022-03-09T22:36:53.620Z]   File "/builds/worker/checkouts/gecko/python/mach/mach/site.py", line 762, in _up_to_date
[task 2022-03-09T22:36:53.620Z]     self._topsrcdir, pthfile_lines, self._site_name, self._requirements
[task 2022-03-09T22:36:53.620Z]   File "/builds/worker/checkouts/gecko/python/mach/mach/site.py", line 1095, in _assert_pip_check
[task 2022-03-09T22:36:53.620Z]     f'The "{virtualenv_name}" site is not compatible with the installed '
[taskcluster 2022-03-09 22:36:54.481Z] === Task Finished ===
[taskcluster 2022-03-09 22:36:54.481Z] Unsuccessful task run with exit code: 1 completed in 606.885 seconds

Comment 10

[Sandor Molnar[:smolnar]]

Backout merged to central

Backout link: https://hg.mozilla.org/mozilla-central/rev/2f742679df4c76ad870a855de3b508efd5ef8843

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0f7ac3c7df38
Expose config variable for Mach native dependency source r=ahal
https://hg.mozilla.org/integration/autoland/rev/d64efc745898
Add "common" to PIP_NETWORK_INSTALL_RESTRICTED_VIRTUALENVS r=ahal

Comment 12

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/0f7ac3c7df38
https://hg.mozilla.org/mozilla-central/rev/d64efc745898

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/95829333db34
Explicitly use system Python packages for Firefox build CI r=ahal

Comment 14

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/95829333db34

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a145943cad60
Remove "use system Python packages" inference r=ahal

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9eb371351104
Simplify "site.py" handling of command site package source r=ahal

Comment 17

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/a145943cad60

Comment 18

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/9eb371351104

Comment 19

[Mathew Hodson]

Can this be closed?

Comment 20

[Mitchell Hentges [:mhentges] 🦀]

Yep, thanks :)