In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the
virtualenvs that we use for building, testing, and other dev tasks. Specifically:
With the push to use
glean for telemetry in
mach, we are requiring (or rather, strongly encouraging) the
glean_sdk Python package to be installed with bug 1651424.
mach bootstrap upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for
mach to function, requiring that the system Python be configured with a certain version of
glean is an unfortunate change.
The build uses the vendored
glean_parser for a number of build tasks. Since the vendored
glean_parser conflicts with the globally-installed
glean_sdk package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781.
We begin to rely more and more on the
zstandard package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's
virtualenv to the SYSTEM
python3 binary, assuming that the system
As we rely more on
zstandard, and other packages that are not vendorable, we need to settle on a standard model for how
mach, the build process, and other
mach commands that may make their own
virtualenvs work in the presence of unvendorable packages.
With that in mind, this patch does all the following:
Separate out the
virtualenv_packages from the in-build
virtualenv_packages. Refactor the common stuff into
common_virtualenv_packages.txt. Add functionality to the
virtualenv_packages manifest parsing to allow the build
virtualenv to "inherit" from the parent by pointing to the parent's
in-virtualenv feature from bug 1655781 is no longer necessary, so delete it.
Add code to
bootstrap, as well as a new
create-mach-environment to create
Add code to
mach to dispatch either to the in-
virtualenvs (or to the system Python 3 for commands which cannot run in the
Remove the "add global argument" feature from
mach. It isn't used and conflicts with (3).
--print-command feature from
mach which is obsoleted by these changes.
This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global
virtualenvs) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes:
We allow users to force running
mach with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these
virtualenvs when they're allowed to continue on with the old behavior if they like. We also skip this in CI.
We needed to duplicate the global-argument logic into the
mach script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the
--print-command feature, justifying its use by saying it was only temporarily required until all
mach commands were running with Python 3. With this change, we'll need to be able to determine the
mach command from the shell script for the forseeable future, and committing to this forever with the cost that
--print-command incurs (namely
mach startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code.