1966763 - Bump Mach's minimum required Python version to 3.9

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, task, P2)

Description

[Alex Hochheiden [:ahochheiden] [he/him]]

Ubuntu 20.04 ends its LTS on May 31st 2025, and I believe that's our last blocker holding us off of Python 3.9 as the minimum for mach.

Comment 1

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1966763 - Bump `mach` minimum `Python` version to `3.9` r?#build

Comment 2

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1966763 - Run `./mach vendor python --upgrade` with the new minimum Python version `3.9` to establish a new baseline r?ahal

Comment 3

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1966763 - CI/Toolchain updates for Python `3.9` r?#build

The Python toolchain we build is now version 3.9.13.

Comment 4

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1966763 - Patch `_sysconfigdata*.py` to use `sys.base_prefix` over "/python" r?#build

This ensures that at runtime the interpreter points at its
actual installed location rather than the fixed staging path,
making it fully relocatable.

Comment 5

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1966763 - Bump `sphinx-js` to `4.0.0` in `docs.txt` for compatibility with `markupsafe` version `3.0.2` r?ahal

Since markupsafe got implicitly bumped in an earlier patch in
this stack, we need to upgrade sphinx-js to stay compatible.

Comment 6

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1966763 - Fix `UP035` and `UP033` `ruff` warnings caused by Python `3.9` version bump r?#build,#linter-reviewers,ahal

Some were automatic fixes, but most were done manually.

Comment 7

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1966763 - `wpt` related Python `3.8` to `3.9` compatibility changes r?jgraham

Comment 8

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1966763 - Check for Python `3.10` or greater r?jgraham

(3, 9, 13) evalutes to greater than (3, 9) which is probably not what was intended.

Comment 9

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1966763 - Remove `lru_cache` from `python.py` to fix `ubuntu1804` test jobs r?ahal

The ubuntu1804 jobs still run with Python 3.8, but ruff now
warns for the old style lru_cache usage with the py39 minimum.

Let's just remove the usage here since it's rather low value anyway.

Comment 10

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1966763 - Bump vendored `pip`, `setuptools`, and `wheel` r?ahal

Updated pip v24.0 -> v25.0.1
Updated setuptools v74.0.0 -> v75.3.1
Updated wheel v0.43.0 -> v0.45.1

Also updated the wheels of these vendored in third_party/python/_venv/wheels/ to match.

The versions for these are the last to support Python 3.8, which we still
need in CI for ubuntu1804 jobs.

Comment 11

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1966763 - Bump `packaging` to version `25.0` r?ahal

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/a21b59136f21
https://hg.mozilla.org/integration/autoland/rev/a007ef628082
CI/Toolchain updates for Python `3.9` r=firefox-build-system-reviewers,glandium
https://github.com/mozilla-firefox/firefox/commit/24282e1a7c0f
https://hg.mozilla.org/integration/autoland/rev/1d98a8f29960
Remove `lru_cache` from `python.py` to fix `ubuntu1804` test jobs r=ahal
https://github.com/mozilla-firefox/firefox/commit/402ee7c138c0
https://hg.mozilla.org/integration/autoland/rev/788d003794fa
Fix `UP035` and `UP033` `ruff` warnings caused by Python `3.9` version bump r=ahal,firefox-build-system-reviewers,linter-reviewers,taskgraph-reviewers,releng-reviewers,translations-reviewers,omc-reviewers,glandium,jcristau,pdahiya,gregtatum
https://github.com/mozilla-firefox/firefox/commit/3265db3f42be
https://hg.mozilla.org/integration/autoland/rev/8e8d0c01039a
Bump `mach` minimum `Python` version to `3.9` r=firefox-build-system-reviewers,mach-reviewers,jgraham,glandium,ahal
https://github.com/mozilla-firefox/firefox/commit/b5d6f4a0fb39
https://hg.mozilla.org/integration/autoland/rev/a9c244e6e845
Patch `_sysconfigdata*.py` to use `sys.base_prefix` over "/python" r=firefox-build-system-reviewers,glandium
https://github.com/mozilla-firefox/firefox/commit/0448774cb7b7
https://hg.mozilla.org/integration/autoland/rev/ec3e0f176425
Run `./mach vendor python --upgrade` with the new minimum Python version `3.9` to establish a new baseline r=firefox-build-system-reviewers,mach-reviewers,glandium,ahal
https://github.com/mozilla-firefox/firefox/commit/523a9ed5dfef
https://hg.mozilla.org/integration/autoland/rev/c9b573f0c619
Bump `sphinx-js` to `4.0.0` in `docs.txt` for compatibility with `markupsafe` version `3.0.2` r=ahal
https://github.com/mozilla-firefox/firefox/commit/077dca7ada08
https://hg.mozilla.org/integration/autoland/rev/479027a3118e
`wpt` related Python `3.8` to `3.9` compatibility changes r=jgraham
https://github.com/mozilla-firefox/firefox/commit/bbb73b3f9058
https://hg.mozilla.org/integration/autoland/rev/448806b71f51
Check for Python `3.10` or greater r=jgraham
https://github.com/mozilla-firefox/firefox/commit/6b25293f06ef
https://hg.mozilla.org/integration/autoland/rev/979617bcae07
Bump vendored `pip`, `setuptools`, and `wheel` r=ahal,mach-reviewers
https://github.com/mozilla-firefox/firefox/commit/08f417ff77ba
https://hg.mozilla.org/integration/autoland/rev/810039b16c9a
Bump `packaging` to version `25.0` r=ahal,mach-reviewers

Comment 13

[amarc]

https://hg.mozilla.org/mozilla-central/rev/a007ef628082
https://hg.mozilla.org/mozilla-central/rev/1d98a8f29960
https://hg.mozilla.org/mozilla-central/rev/788d003794fa
https://hg.mozilla.org/mozilla-central/rev/8e8d0c01039a
https://hg.mozilla.org/mozilla-central/rev/a9c244e6e845
https://hg.mozilla.org/mozilla-central/rev/ec3e0f176425
https://hg.mozilla.org/mozilla-central/rev/c9b573f0c619
https://hg.mozilla.org/mozilla-central/rev/479027a3118e
https://hg.mozilla.org/mozilla-central/rev/448806b71f51
https://hg.mozilla.org/mozilla-central/rev/979617bcae07
https://hg.mozilla.org/mozilla-central/rev/810039b16c9a

Comment 14

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/53154 for changes under testing/web-platform/tests

Comment 15

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I'm stumbling over this landing now when trying to pull from the Firefox repository via git. The reason is most likely the rename of the package name for MarkupSafe to markupsafe. It's telling me that The following untracked working tree files would be overwritten by merge but this folder exists. This is an issue I've hit a couple of times meanwhile and for now I've set git config core.ignorecase false which at least seems to help.

Glandium, there are a couple of folks having the same problem (recently discussed in #jj) so we wonder what's missing in our git config for Mozilla to not be affected here.

Comment 17

[Mike Hommey [:glandium]]

I don't know. Things work properly on my mac without touching anything.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Mike Hommey [:glandium] from comment #17)

I don't know. Things work properly on my mac without touching anything.

So I assume you don't have a case-sensitive file system? In case of my APFS partition (as created by default via the macOS installer) it is case-insensitive and as such treats file and File as the same. Doing just a rename of the file will put folks with those file systems into trouble. As such it's adviced to not just rename in one step but (as it looks like) have a two-step rename:

git mv -f File.txt tempfile.txt
git mv tempfile.txt file.txt
git commit -m "Rename with case change workaround"

Alex, if more renames need to be done maybe you can check that? It's a really annoying situation right now for me given that I seem to have to re-create all my old branches which I'm not able to rebase against main.

Comment 19

[Mike Hommey [:glandium]]

(In reply to Henrik Skupin [:whimboo][⌚️UTC+2] from comment #18)

(In reply to Mike Hommey [:glandium] from comment #17)

I don't know. Things work properly on my mac without touching anything.

So I assume you don't have a case-sensitive file system? In case of my APFS partition (as created by default via the macOS installer) it is case-insensitive and as such treats file and File as the same.

My APFS is case insensitive:

% ls third_party/python/markupsaFe 
CHANGES.rst	MANIFEST.in	pyproject.toml	requirements	setup.py	tox.ini
LICENSE.txt	PKG-INFO	README.md	setup.cfg	src
% ls third_party/python/markupsafe
CHANGES.rst	MANIFEST.in	pyproject.toml	requirements	setup.py	tox.ini
LICENSE.txt	PKG-INFO	README.md	setup.cfg	src

Doing just a rename of the file will put folks with those file systems into trouble. As such it's adviced to not just rename in one step but (as it looks like) have a two-step rename:

git mv -f File.txt tempfile.txt
git mv tempfile.txt file.txt
git commit -m "Rename with case change workaround"

This only works to do a rename locally. That won't change anything if the history itself is creating a confusion on your system, since it will lead to a single commit. But really, I don't see what's different between our setups. On the the other hand, you only said "when trying to pull from the Firefox repository via git", which doesn't say much...

Comment 20

[Alex Hochheiden [:ahochheiden] [he/him]]

I'm not sure if there's anything I can do here.

I will note that there's one more rename coming from bug 1972440. That will be the last one from third_party/python. I think there was a PEP around the time of Python 3.8 -> Python 3.9 that standardized on lowercase directories, so now that we've bumped to Python 3.9 as the minimum we've hit all these renames.

It's a bit annoying for jj too, but with that you can just jj abandon and it fixes itself. It always shows up if you cross that revision though. IIRC there's a bug upstream for jj to fix this, but maybe the same is necessary for git?

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Mike Hommey [:glandium] from comment #19)

My APFS is case insensitive:

% ls third_party/python/markupsaFe 
CHANGES.rst	MANIFEST.in	pyproject.toml	requirements	setup.py	tox.ini
LICENSE.txt	PKG-INFO	README.md	setup.cfg	src
% ls third_party/python/markupsafe
CHANGES.rst	MANIFEST.in	pyproject.toml	requirements	setup.py	tox.ini
LICENSE.txt	PKG-INFO	README.md	setup.cfg	src

Sure and that works fine for me as well. My question was around git. Can you easily checkout commits from recent main and a changeset before all these changes on this bug? You will have to delete the files first. Then also create a new branch from such an older commit, add some extra commits on top and then try to rebase it against main. I cannot see how to correctly do the latter action to get my older commits rebased at the moment. Happy to file a separate bug about that if wanted.

Comment 22

[Mike Hommey [:glandium]]

I'd suggest starting a thread on #engineering-workflow on slack with clear steps to reproduce first.