1751663 - Build configuration fails with MACH_USE_SYSTEM_PYTHON when zstandard-0.17.0 is found

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, defect, P3)

Description

[Joonas Niilola]

Steps to reproduce:

Install pip,
install the latest zstandard-0.17.0,
try to build Firefox with the following defined:
# Use system's Python environment
export MACH_USE_SYSTEM_PYTHON=1
export MACH_SYSTEM_ASSERTED_COMPATIBLE_WITH_MACH_SITE=1
export PIP_NO_CACHE_DIR=off

Actual results:

Configuring source in /var/tmp/portage/www-client/firefox-96.0.2/work/firefox-96.0.2 ...

• Current BINDGEN_CFLAGS: no value set
• Current CFLAGS: -march=native -O2 -pipe
• Current CXXFLAGS: -march=native -O2 -pipe
• Current LDFLAGS: -Wl,-O1 -Wl,--as-needed
• Current RUSTFLAGS: no value set
• Enforcing the use of clang due to USE=clang ...
• Build BINDGEN_CFLAGS: no value set
• Build CFLAGS: -march=native -pipe
• Build CXXFLAGS: -march=native -pipe
• Build LDFLAGS: -Wl,-O1 -Wl,--as-needed -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags
• Build RUSTFLAGS: no value set

==========================================================
Building firefox-96.0.2 with the following configuration
--enable-application=browser mozilla.org default
--allow-addon-sideload Gentoo default
--disable-cargo-incremental Gentoo default
--disable-crashreporter Gentoo default
--disable-install-strip Gentoo default
--disable-strip Gentoo default
--disable-updater Gentoo default
--enable-official-branding Gentoo default
--enable-release Gentoo default
--enable-system-ffi Gentoo default
--enable-system-pixman Gentoo default
--host=x86_64-pc-linux-gnu Gentoo default
--libdir=/usr/lib64 Gentoo default
--prefix=/usr Gentoo default
--target=x86_64-pc-linux-gnu Gentoo default
--without-ccache Gentoo default
--without-wasm-sandboxed-libraries Gentoo default
--with-intl-api Gentoo default
--with-libclang-path=/usr/lib/llvm/13/lib64 Gentoo default
--with-system-nspr Gentoo default
--with-system-nss Gentoo default
--with-system-zlib Gentoo default
--with-toolchain-prefix=x86_64-pc-linux-gnu- Gentoo default
--with-unsigned-addon-scopes=app,system Gentoo default
--x-includes=/usr/include Gentoo default
--x-libraries=/usr/lib64 Gentoo default
--update-channel=release mozilla.org default
--enable-rust-simd mozilla.org default
--with-google-safebrowsing-api-keyfile=/var/tmp/portage/www-client/firefox-96.0.2/work/firefox-96.0.2/api-google.key Gentoo default
--with-google-location-service-api-keyfile=/var/tmp/portage/www-client/firefox-96.0.2/work/firefox-96.0.2/api-location.key Gentoo default
--with-mozilla-api-keyfile=/var/tmp/portage/www-client/firefox-96.0.2/work/firefox-96.0.2/api-mozilla.key Gentoo default
--without-system-av1 -system-av1
--with-system-harfbuzz +system-harfbuzz
--with-system-graphite2 +system-harfbuzz
--with-system-icu +system-icu
--with-system-jpeg +system-jpeg
--with-system-libevent=/usr +system-libevent
--with-system-libvpx +system-libvpx
--without-system-png -system-png
--with-system-webp +system-webp
--enable-dbus +dbus
--disable-geckodriver -geckodriver
--disable-jack -jack
--disable-pulseaudio -pulseaudio
--enable-alsa -pulseaudio
--disable-sndio -sndio
--disable-necko-wifi -wifi
--enable-default-toolkit=cairo-gtk3 mozilla.org default
--enable-linker=lld forcing ld=lld due to USE=clang
--disable-debug -debug
--disable-debug-symbols Gentoo default
--enable-optimize=-O2 Gentoo default
--disable-elf-hack elf-hack is broken when using Clang
XARGS=/usr/bin/xargs Gentoo default

zstandard<=0.16.0,>=0.11.1: Installed with unexpected version "0.17.0"
Traceback (most recent call last):
File "/var/tmp/portage/www-client/firefox-96.0.2/work/firefox-96.0.2/./mach", line 126, in <module>
main(sys.argv[1:])
File "/var/tmp/portage/www-client/firefox-96.0.2/work/firefox-96.0.2/./mach", line 118, in main
mach = check_and_get_mach(os.path.dirname(os.path.realpath(file)))
File "/var/tmp/portage/www-client/firefox-96.0.2/work/firefox-96.0.2/./mach", line 105, in check_and_get_mach
return load_mach(dir_path, mach_path)
File "/var/tmp/portage/www-client/firefox-96.0.2/work/firefox-96.0.2/./mach", line 91, in load_mach
return mach_initialize.initialize(dir_path)
File "/var/tmp/portage/www-client/firefox-96.0.2/work/firefox-96.0.2/build/mach_initialize.py", line 219, in initialize
_activate_python_environment(
File "/var/tmp/portage/www-client/firefox-96.0.2/work/firefox-96.0.2/build/mach_initialize.py", line 172, in _activate_python_environment
mach_environment = MachSiteManager.from_environment(
File "/var/tmp/portage/www-client/firefox-96.0.2/work/firefox-96.0.2/python/mach/mach/site.py", line 259, in from_environment
if external_python.provides_any_package("mach", requirements)
File "/var/tmp/portage/www-client/firefox-96.0.2/work/firefox-96.0.2/python/mach/mach/site.py", line 800, in provides_any_package
raise Exception(
Exception: The Python packages associated with "/usr/bin/python3.10" aren't compatible with the "mach" virtualenv

• ERROR: www-client/firefox-96.0.2::gentoo failed (configure phase):
• (no error message)

Expected results:

I had the impression zstandard was an optional dependency, so the build shouldn't fail regardless whether a correct version is not found or not. And this only fails with pip installed. If pip is not installed, then the configuration (and building) will happily continue and work.

For now I've simply removed the "raise Exception" part,
diff -Naur a/python/mach/mach/site.py b/python/mach/mach/site.py
--- a/python/mach/mach/site.py 2022-01-20 00:44:27.000000000 +0200
+++ b/python/mach/mach/site.py 2022-01-24 08:45:34.705599229 +0200
@@ -797,10 +797,6 @@

     if not result.has_all_packages:
         print(result.report())

•        raise Exception(
  

•            f'The Python packages associated with "{self.python_path}" aren\'t '
  

•            f'compatible with the "{virtualenv_name}" virtualenv'
  

•        )
  
     return result.provides_any_package
  

but I know it's not the right way to fix this.

Firefox-96.0.2 with newly published zstandard.
https://pypi.org/project/zstandard/#history

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::Mach Core' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

We'll work around this by bumping the maximum zstandard version to 0.17.0.

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1751663: Bump maximum zstandard version to `0.17.0`

It doesn't appear like the breaking changes will affect us.

Note that the taskcluster packages config wasn't updated because
0.17.0 hasn't yet been deployed to the GitHub releases page.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/64e88f42fea6
Bump maximum zstandard version to `0.17.0` r=firefox-build-system-reviewers,glandium

Comment 5

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/64e88f42fea6