1935621 - mach fails with Python 3.12.8, Python 3.13.1 (regression with new Python releases)

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Sam James]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0

Steps to reproduce:

1. Install Python 3.12.8 or Python 3.13.1 (3.12.7, 3.13.0 are fine)
2. Unpack firefox-133.0.source.tar.xz
3. Run ./mach

Actual results:

$ ./mach
Traceback (most recent call last):
  File "/tmp/ff2/firefox-133.0/./mach", line 155, in main
    mach = check_and_get_mach(os.path.dirname(os.path.realpath(__file__)), args)
  File "/tmp/ff2/firefox-133.0/./mach", line 39, in check_and_get_mach
    return load_mach(dir_path, mach_path, args)
  File "/tmp/ff2/firefox-133.0/./mach", line 25, in load_mach
    return mach_initialize.initialize(dir_path, args)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/tmp/ff2/firefox-133.0/build/mach_initialize.py", line 344, in initialize
    command_site_manager.activate()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/tmp/ff2/firefox-133.0/python/mach/mach/site.py", line 602, in activate
    self.ensure()
    ~~~~~~~~~~~^^
  File "/tmp/ff2/firefox-133.0/python/mach/mach/site.py", line 563, in ensure
    result = self._up_to_date()
  File "/tmp/ff2/firefox-133.0/python/mach/mach/site.py", line 800, in _up_to_date
    pthfile_lines = self._pthfile_lines()
  File "/tmp/ff2/firefox-133.0/python/mach/mach/site.py", line 782, in _pthfile_lines
    PythonVirtualenv(self._mach_virtualenv_root).site_packages_dirs()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/tmp/ff2/firefox-133.0/python/mach/mach/site.py", line 853, in site_packages_dirs
    platlib = self.resolve_sysconfig_packages_path("platlib")
  File "/tmp/ff2/firefox-133.0/python/mach/mach/site.py", line 843, in resolve_sysconfig_packages_path
    relative_path = path.relative_to(data_path)
  File "/usr/lib/python3.13/pathlib/_local.py", line 385, in relative_to
    raise ValueError(f"{str(self)!r} is not in the subpath of {str(other)!r}")
ValueError: '/usr/lib/python3.13/site-packages' is not in the subpath of '/home/sam/.mozbuild/srcdirs/firefox-133.0-bd1eb2302b3c/_virtualenvs/mach'

---

Note that you are running Mach with Python 3.13.1, which is higher
than the highest known working version of Python for Mach. Consider
running Mach with Python 3.11 or lower.

Running the following command may solve your issue:

    python3.10 ./mach

Expected results:

Successful build (no Python pathlib errors).

Comment 1

[Sam James]

I'll probably bisect CPython in a moment.

Comment 2

[Sam James]

This regressed with the following commit in Python 3.12.8:

commit 50e42b99802882d5e1f6286ec988ed310370a6ff
Author: Miss Islington (bot) <[e-mail]>
Date: Sun Nov 17 01:26:59 2024 +0100

[3.12] GH-126789: fix some sysconfig data on late site initializations

Co-authored-by: Filipe Laíns 🇵🇸[e-mail]>

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::General' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 4

[Sam James]

... and mentioned at https://github.com/python/cpython/issues/126789#issuecomment-2522791287 for advice.

Comment 5

[Filipe Laíns]

Attachment: Bug 1935621 - Fix virtual environment sysconfig path calculation

Signed-off-by: Filipe Laíns <[e-mail]>

Comment 6

[Filipe Laíns]

FYI, this issue should trigger with Python >=3.12.8, and >=3.13.1. It shouldn't trigger on 3.14 though, since I made some further changes that fixed the underlying issue that motivated the bugfix, so it was reverted as no longer necessary, but I didn't take into account users doing the kind of thing I encountered here (I expand on this below).

The current code makes some assumptions on the Python install layout, which are fragile, but the source of the issue was the activate_virtualenv function below overwriting sys.prefix (please don't do this, it's very fragile!).

https://github.com/mozilla/gecko-dev/blob/a77c79d60e2adee009b8641b3a80d9520481c01a/python/mach/mach/site.py#L1391

(more context below, feel free to skip if you aren't interested, it's not required to understand this issue)

What triggered this error was a change I made in the upstream that fixed sysconfig.get_config_vars — and by extension sysconfig.get_paths — returning outdated cached data if sys.prefix/sys.exec_prefix were changed after the sysconfig import, which activate_virtualenv does.
This was originally motivated by the fact that virtual environment activation, which updates sys.prefix/sys.exec_prefix, was implemented in the site module. That meant that running Python with -S resulted in the virtual environment not being activated at startup, and calling site.main() would activate the virtual environment in the middle of execution. What motivated the sysconfig change was the fact that since we cached the sys.prefix/sys.exec_prefix values during import, in the mentioned scenario, the sysconfig data would differ depending on whether sysconfig was imported before or after site.main() was called — if before, the data would be incorrect.
After fixing the caching issue, we moved the virtual environment activation on 3.14 from site to the interpreter path initialization, meaning sys.prefix/sys.exec_prefix will no longer change after initializing. I then reverted the cache bugfix since the underlying issue got fixed, but I did not consider people manually updating sys.prefix/sys.exec_prefix, which is unsupported but also triggers the issue. Because of this, I think I am gonna re-apply the bugfix.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b0dbc944fb7b
Fix virtual environment sysconfig path calculation r=firefox-build-system-reviewers,ahochheiden

Comment 8

[amarc]

https://hg.mozilla.org/mozilla-central/rev/b0dbc944fb7b

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:lains, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox134 to wontfix.

For more information, please visit BugBot documentation.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9442305 [details]
Bug 1935621 - Fix virtual environment sysconfig path calculation

Beta/Release Uplift Approval Request

• User impact if declined/Reason for urgency: No end user impact, just pain for developers & downstreams trying to build and test Firefox locally.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce: n/a
• List of other uplifts needed: n/a
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): No end user impact, CI still still works :)
• String changes made/needed: none
• Is Android affected?: No

Comment 11

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/24bcc88ae0e8

Comment 12

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr128/rev/e270c9ac0dbb