1717884 - mach build using the system Python fails to bootstrap because it cannot find the zstandard Python module

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, defect, P5)

Description

[Chris Siebenmann]

On Fedora 33 x86_64, I have recently stopped being able to build Firefox trunk (some time since the 21st, the last time I'm sure I did this) because the bootstrap environment can't find the Python zstandard module. I have the module installed.

The error is:

 2:17.29 Error running mach:
 2:17.29     ['--log-no-times', 'artifact', 'toolchain', '--from-build', 'toolchain-linux64-clang']
 2:17.29 The error occurred in code that was called by the mach command. This is either
 2:17.29 a bug in the called code itself or in the way that mach is calling it.
 2:17.29 You can invoke |./mach busted| to check if this issue is already on file. If it
 2:17.29 isn't, please use |./mach busted file artifact| to report it. If |./mach busted| is
 2:17.29 misbehaving, you can also inspect the dependencies of bug 1543241.
 2:17.29 If filing a bug, please include the full output of mach, including this error
 2:17.29 message.
 2:17.29 The details of the failure are as follows:
 2:17.29 ModuleNotFoundError: No module named 'zstandard'
 2:17.29   File "/data/code/mozilla-hg/mozilla-central/python/mozbuild/mozbuild/artifact_commands.py", line 573, in artifact_toolchain
 2:17.29     unpack_file(local)
 2:17.29   File "/data/code/mozilla-hg/mozilla-central/python/mozbuild/mozbuild/action/tooltool.py", line 993, in unpack_file
 2:17.29     import zstandard
 2:17.35 ERROR: Command '['/usr/bin/sh', '/data/code/mozilla-hg/mozilla-central/mach', '--log-no-times', 'artifact', 'toolchain', '--from-build', 'toolchain-linux64-clang']' returned non-zero exit status 1.
 2:17.35 ERROR: If you can't fix the above, retry with --disable-bootstrap.
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file build| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 1: ['/usr/bin/python3', '/data/code/mozilla-hg/mozilla-central/configure.py']

  File "/data/code/mozilla-hg/mozilla-central/python/mozbuild/mozbuild/build_commands.py", line 159, in build
    return driver.build(
  File "/data/code/mozilla-hg/mozilla-central/python/mozbuild/mozbuild/controller/building.py", line 1144, in build
    config_rc = self.configure(
  File "/data/code/mozilla-hg/mozilla-central/python/mozbuild/mozbuild/controller/building.py", line 1528, in configure
    status = self._run_command_in_objdir(
  File "/data/code/mozilla-hg/mozilla-central/python/mozbuild/mozbuild/base.py", line 845, in _run_command_in_objdir
    return self.run_process(cwd=self.topobjdir, **args)
  File "/data/code/mozilla-hg/mozilla-central/python/mach/mach/mixin/process.py", line 176, in run_process
    raise Exception(

I'm running mach build as: MACH_USE_SYSTEM_PYTHON=1 TMP=() TMPDIR=() MOZ_NOSPAM=1 ./mach build, working with a completely clean objdir. I haven't tested if this happens when not using the system Python.

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

It's because of MACH_USE_SYSTEM_PYTHON=1 - we install the zstandard module when we get to use our own virtualenvs, and you're currently working around it.

If you're sure that you want to use the system python (can you leave a comment as to why?) then you can work around the issue with pip3 install zstandard==0.15.2.

Comment 2

[Chris Siebenmann]

But I have zstandard 0.15.2 installed in the system Python:

$ pip3 list | fgrep zstandard
zstandard                0.15.2

It is just that mach build is not picking it up.

I want to use the system Python because I don't feel like having the Mozilla build process chew up extra space in my $HOME to duplicate everything the system Python already has.

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

It is just that mach build is not picking it up.

Hmm, that's bizarre, it should be able to see it. If you run python3, then do import zstandard, does that successfully import?

I want to use the system Python because I don't feel like having the Mozilla build process chew up extra space in my $HOME to duplicate everything the system Python already has.

This is unfortunately the nature of Python - since only version of a package (including transitive) can be installed at a time, we're forced to use virtualenvs to ensure that our python requirements and user's wanted pip-installed tools don't collide.

I'll reopen this ticket, but I won't be able to commit time to resolving it due to it being outside of our main supported workflows.
Feel free to dig into this, though. I'd recommend doing things like:

• If you run ./mach python, then import zstandard, does that work?
• Check the contents of sys.path when import zstandard fails.

Feel free to raise questions here or in #build on Matrix if you have questions about how the internals tick.

Comment 4

[Chris Siebenmann]

The standard sys.path in my Python 3, and a clean import of zstandard:

Python 3.9.5 (default, May 14 2021, 00:00:00) 
[GCC 10.3.1 20210422 (Red Hat 10.3.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib64/python39.zip', '/usr/lib64/python3.9', '/usr/lib64/python3.9/lib-dynload', '/homes/hawklords/cks/.local/lib/python3.9/site-packages', '/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages']
>>> import zstandard
>>>

The zstandard I'm using is the Fedora 33 python3-zstandard package, which puts things in /usr/lib64/python3.9/site-packages/zstandard/. Based on examining various things, I think that the virtual environment in the build objdir is created without using --system-site-packages.

Running MACH_USE_SYSTEM_PYTHON=1 ./mach python and then importing zstandard fails. The sys.path here is:

Python 3.9.5 (default, May 14 2021, 00:00:00) 
[GCC 10.3.1 20210422 (Red Hat 10.3.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import zstandard
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'zstandard'
>>> import sys
>>> sys.path
['', '/usr/lib64/python39.zip', '/usr/lib64/python3.9', '/usr/lib64/python3.9/lib-dynload', '/data/code/mozilla-hg/build-objdir/_virtualenvs/common/lib64/python3.9/site-packages', '/data/code/mozilla-hg/build-objdir/_virtualenvs/common/lib/python3.9/site-packages', '/data/code/mozilla-hg/mozilla-central/build', '/data/code/mozilla-hg/mozilla-central/config', '/data/code/mozilla-hg/mozilla-central/config/mozunit', '/data/code/mozilla-hg/mozilla-central/dom/bindings', '/data/code/mozilla-hg/mozilla-central/dom/bindings/parser', '/data/code/mozilla-hg/mozilla-central/layout/tools/reftest', '/data/code/mozilla-hg/mozilla-central/python/l10n', '/data/code/mozilla-hg/mozilla-central/python/mach', '/data/code/mozilla-hg/mozilla-central/python/mozboot', '/data/code/mozilla-hg/mozilla-central/python/mozbuild', '/data/code/mozilla-hg/mozilla-central/python/mozlint', '/data/code/mozilla-hg/mozilla-central/python/mozperftest', '/data/code/mozilla-hg/mozilla-central/python/mozrelease', '/data/code/mozilla-hg/mozilla-central/python/mozterm', '/data/code/mozilla-hg/mozilla-central/python/mozversioncontrol', '/data/code/mozilla-hg/mozilla-central/taskcluster', '/data/code/mozilla-hg/mozilla-central/testing', '/data/code/mozilla-hg/mozilla-central/testing/condprofile', '/data/code/mozilla-hg/mozilla-central/testing/firefox-ui/harness', '/data/code/mozilla-hg/mozilla-central/testing/marionette/client', '/data/code/mozilla-hg/mozilla-central/testing/marionette/harness', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/manifestparser', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozcrash', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozdebug', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozdevice', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozfile', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozhttpd', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozgeckoprofiler', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozinfo', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozinstall', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozleak', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozlog', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/moznetwork', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozpower', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozprocess', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozprofile', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozproxy', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozrunner', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozsystemmonitor', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozscreenshot', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/moztest', '/data/code/mozilla-hg/mozilla-central/testing/mozbase/mozversion', '/data/code/mozilla-hg/mozilla-central/testing/raptor', '/data/code/mozilla-hg/mozilla-central/testing/talos', '/data/code/mozilla-hg/mozilla-central/testing/web-platform', '/data/code/mozilla-hg/mozilla-central/testing/web-platform/tests/tools/third_party/certifi', '/data/code/mozilla-hg/mozilla-central/testing/web-platform/tests/tools/third_party/h2', '/data/code/mozilla-hg/mozilla-central/testing/web-platform/tests/tools/third_party/hpack', '/data/code/mozilla-hg/mozilla-central/testing/web-platform/tests/tools/third_party/html5lib', '/data/code/mozilla-hg/mozilla-central/testing/web-platform/tests/tools/third_party/hyperframe', '/data/code/mozilla-hg/mozilla-central/testing/web-platform/tests/tools/third_party/pywebsocket3', '/data/code/mozilla-hg/mozilla-central/testing/web-platform/tests/tools/third_party/webencodings', '/data/code/mozilla-hg/mozilla-central/testing/web-platform/tests/tools/wptserve', '/data/code/mozilla-hg/mozilla-central/testing/web-platform/tests/tools/wptrunner', '/data/code/mozilla-hg/mozilla-central/testing/xpcshell', '/data/code/mozilla-hg/mozilla-central/third_party/python/appdirs', '/data/code/mozilla-hg/mozilla-central/third_party/python/atomicwrites', '/data/code/mozilla-hg/mozilla-central/third_party/python/attrs', '/data/code/mozilla-hg/mozilla-central/third_party/python/blessings', '/data/code/mozilla-hg/mozilla-central/third_party/python/cbor2', '/data/code/mozilla-hg/mozilla-central/third_party/python/chardet', '/data/code/mozilla-hg/mozilla-central/third_party/python/Click', '/data/code/mozilla-hg/mozilla-central/third_party/python/compare_locales', '/data/code/mozilla-hg/mozilla-central/third_party/python/cookies', '/data/code/mozilla-hg/mozilla-central/third_party/python/cram', '/data/code/mozilla-hg/mozilla-central/third_party/python/diskcache', '/data/code/mozilla-hg/mozilla-central/third_party/python/distro', '/data/code/mozilla-hg/mozilla-central/third_party/python/dlmanager', '/data/code/mozilla-hg/mozilla-central/third_party/python/ecdsa', '/data/code/mozilla-hg/mozilla-central/third_party/python/esprima', '/data/code/mozilla-hg/mozilla-central/third_party/python/fluent.migrate', '/data/code/mozilla-hg/mozilla-central/third_party/python/fluent.syntax', '/data/code/mozilla-hg/mozilla-central/third_party/python/funcsigs', '/data/code/mozilla-hg/mozilla-central/third_party/python/gyp/pylib', '/data/code/mozilla-hg/mozilla-central/third_party/python/idna', '/data/code/mozilla-hg/mozilla-central/third_party/python/importlib_metadata', '/data/code/mozilla-hg/mozilla-central/third_party/python/iso8601', '/data/code/mozilla-hg/mozilla-central/third_party/python/Jinja2', '/data/code/mozilla-hg/mozilla-central/third_party/python/jsmin', '/data/code/mozilla-hg/mozilla-central/third_party/python/json-e', '/data/code/mozilla-hg/mozilla-central/third_party/python/jsonschema', '/data/code/mozilla-hg/mozilla-central/third_party/python/MarkupSafe/src', '/data/code/mozilla-hg/mozilla-central/third_party/python/mohawk', '/data/code/mozilla-hg/mozilla-central/third_party/python/more_itertools', '/data/code/mozilla-hg/mozilla-central/third_party/python/mozilla_version', '/data/code/mozilla-hg/mozilla-central/third_party/python/pathspec', '/data/code/mozilla-hg/mozilla-central/third_party/python/pep487/lib', '/data/code/mozilla-hg/mozilla-central/third_party/python/pluggy', '/data/code/mozilla-hg/mozilla-central/third_party/python/ply', '/data/code/mozilla-hg/mozilla-central/third_party/python/py', '/data/code/mozilla-hg/mozilla-central/third_party/python/pyasn1', '/data/code/mozilla-hg/mozilla-central/third_party/python/pyasn1-modules', '/data/code/mozilla-hg/mozilla-central/third_party/python/pylru', '/data/code/mozilla-hg/mozilla-central/third_party/python/pyrsistent', '/data/code/mozilla-hg/mozilla-central/third_party/python/pystache', '/data/code/mozilla-hg/mozilla-central/third_party/python/pytest', '/data/code/mozilla-hg/mozilla-central/third_party/python/python-hglib', '/data/code/mozilla-hg/mozilla-central/third_party/python/pytoml', '/data/code/mozilla-hg/mozilla-central/third_party/python/PyYAML/lib3', '/data/code/mozilla-hg/mozilla-central/third_party/python/redo', '/data/code/mozilla-hg/mozilla-central/third_party/python/requests', '/data/code/mozilla-hg/mozilla-central/third_party/python/requests-unixsocket', '/data/code/mozilla-hg/mozilla-central/third_party/python/responses', '/data/code/mozilla-hg/mozilla-central/third_party/python/rsa', '/data/code/mozilla-hg/mozilla-central/third_party/python/sentry_sdk', '/data/code/mozilla-hg/mozilla-central/third_party/python/six', '/data/code/mozilla-hg/mozilla-central/third_party/python/slugid', '/data/code/mozilla-hg/mozilla-central/third_party/python/taskcluster', '/data/code/mozilla-hg/mozilla-central/third_party/python/taskcluster-urls', '/data/code/mozilla-hg/mozilla-central/third_party/python/typing_extensions', '/data/code/mozilla-hg/mozilla-central/third_party/python/urllib3', '/data/code/mozilla-hg/mozilla-central/third_party/python/voluptuous', '/data/code/mozilla-hg/mozilla-central/third_party/python/yamllint', '/data/code/mozilla-hg/mozilla-central/third_party/python/zipp', '/data/code/mozilla-hg/mozilla-central/toolkit/components/telemetry/tests/marionette/harness', '/data/code/mozilla-hg/mozilla-central/tools', '/data/code/mozilla-hg/mozilla-central/tools/moztreedocs', '/data/code/mozilla-hg/mozilla-central/xpcom/ds/tools', '/data/code/mozilla-hg/mozilla-central/xpcom/idl-parser', '/data/code/mozilla-hg/mozilla-central/third_party/python/glean_parser']
>>> 

Comment 5

[Mike Hommey [:glandium]]

Running MACH_USE_SYSTEM_PYTHON=1 ./mach python and then importing zstandard fails. The sys.path here is:

That doesn't work because mach python always uses the objdir virtualenv python. MACH_USE_SYSTEM_PYTHON only makes mach use the system python when it would otherwise use the mach global virtualenv.

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #1)

If you're sure that you want to use the system python (can you leave a comment as to why?) then you can work around the issue with pip3 install zstandard==0.15.2.

Unfortunately, bug 1703055 explicitly made that not work.

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

Unfortunately, bug 1703055 explicitly made that not work.

Good catch.

---

I'm not sure how confident I am that we should continue to support MACH_USE_SYSTEM_PYTHON, especially as the Python dependency situation gets more complex. However, I'm content to defer removing it until necessary. So, in the interrim, our bug is that MACH_USE_SYSTEM_PYTHON doesn't work due to system packages being excluded from the import path.

So, in the meantime, if anybody is interested and wants to write a patch, you can tweak the system package filter to respect MACH_USE_SYSTEM_PYTHON.

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

Hmm, this use case overlaps pretty strongly with automation (don't create virtualenvs or install from PyPI, generally), so I've resolved this in an upcoming patch for 1717051.

The new behaviour will be that system-package-isolation will only occur if we're able to create and manage our own virtualenvs. If not, then we'll attempt to source imports from the system. Note that version validation doesn't (yet?) occur, so if incorrect versions of psutil/zstandard/glean_sdk are installed, then unexpected errors may occur.

Comment 9

[Mitchell Hentges [:mhentges] 🦀]

This should be resolved now, though please re-open if this assumption is incorrect.

One surprise here is that, even if MACH_USE_SYSTEM_PYTHON=1, we still use the Mach virtualenv to do ./mach artifact toolchain during ./mach bootstrap.
So, reading back through this bug, I'm guessing that zstandard failed to install when it did the mach-virtualenv-creation?

Either way, we no longer isolate from the system whenMACH_USE_SYSTEM_PYTHON=1, so zstandard should import nicely.