1734402 - Bump Mach's minimum required Python version to 3.7

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement, P3)

Description

[Mitchell Hentges [:mhentges] 🦀]

Python 3.7 has some cool new features, some of which that I'm most excited about are:

• dataclasses
• postponed typing annotation evaluation

Python 3.6 is end-of-life'd on December 23 this year.

---

One question here is whether downstream Firefox builders (e.g. old Debian, Ubuntu, CentOS, etc) can provide Python 3.7 for the build, or if they're only running Python 3.6 with their own security patches. Investigation is needed.

Comment 1

[Olivier Tilloy]

Ubuntu's policy on stable release updates is to apply selected security patches rather than update to a new version.
Ubuntu 18.04, generally supported until April 2023, has Python 3.6.7.

That release is scheduled to continue receiving security updates for another 5 years (until 2028), but in theory this excludes applications like Firefox, for which the official recommendation is to install the snap package (which itself is built on a more recent base, so not affected by this problem).

Ubuntu 16.04 is already out of the equation for the same reason, i.e. we're not building Firefox there any longer.
When it was still built there (until April 2021), we had to carry a set of distro-patches to revert some Python 3.6 specifics (because 16.04 only had 3.5.1):

https://bazaar.launchpad.net/~mozillateam/firefox/firefox.xenial/view/head:/debian/patches/lower-python3-requirement.patch
https://bazaar.launchpad.net/~mozillateam/firefox/firefox.xenial/view/head:/debian/patches/python3-remove-variable-annotations.patch
https://bazaar.launchpad.net/~mozillateam/firefox/firefox.xenial/view/head:/debian/patches/python3-remove-fstrings.patch
https://bazaar.launchpad.net/~mozillateam/firefox/firefox.xenial/view/head:/debian/patches/python3-remove-pep487.patch

When the requirement is bumped to Python 3.7, we'll likely have to do something similar to continue building Firefox on Ubuntu 18.04 until April 2023.

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Tracking impact to other distros:

• I received an email back from Martin Stránský from Red Hat, and it sounds like RHEL 8 is able to build Firefox with Python 3.8 because it's in "modules".

---

Hmm, that makes sense, thanks Olivier.
To confirm, it sounds like "Standard Support" vs "Extended Security Maintenance" handles applications like Firefox differently:

• In "Standard Support": Firefox is still built using packages associated with the current Ubuntu release, and is itself available as an official apt-installable package. Installing via snap is also allowed.
• In "Extended Security Maintenance", Firefox is no longer built or available as an official apt-installable package, but can only be installed via snap.
  • Unrelated, but I wonder what happens if the firefox package is installed in 18.04 when it ticks over to "Extended Security Maintenance", is it automatically uninstalled and replaced with the associated snap package?

---

Hmm, this will require some further pondering then, since you have to patch out all Python 3.7 usages, which I can understand to be annoying.
I bet the most impactful feature of Python 3.7 that will be used is dataclasses, which would require simple but boilerplate-y __init__() creation.
There's also the various new parameters and function changes that would require backports. Fortunately, they should mostly all "fail fast" (at call-time), but may not be trivial to fix.

One option here is that, with the recent improvements to Python dependency-management, we might be able to do something flexible like allow specific Mach commands to use newer Python versions, while still forcing Mach and the build to be compatible with Python 3.6.

I'm going to chew on this, thanks :)

Comment 3

[James Graham [:jgraham]]

There is some pressure to update wpt to 3.7+ because various dependencies (like the HTTP/3 server) are only releasing updates for 3.7+. I assume Ubuntu don't need to be able to run tests, so if we could update test jobs in CI to 3.7, that would be very welcome, even if the build itself continues to support 3.6 for another year.

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

I've created a ticket to track that here.
Note that, if you're bumping your vendored packages to be Python 3.7+, then you'll need to tweak Mach to not depend on them (moving them to a WPT-specific site/virtualenv).

Comment 5

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1734402 - Upgrade to Python 3.7 for Ubuntu1804 r?#build

Comment 6

[James Graham [:jgraham]]

Just as a further note: wpt is now in the position where we're having to patch third party libraries we've vendored in to get both Python 3.10 and 3.6 support, and has stopped taking almost any dependency updates because the ecosystem seems to be fully committed to 3.7+ as the targets. I think the local-only deployment of wpt, and lack of untrusted input, makes it less likely that any patched security issues in older library versions are realistically a problem, but it's at least uncomfortable.

Comment 7

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1734402 - Bump Mach's minimum Python version to 3.7 r?#build

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e1f91aaed4c3
Upgrade to Python 3.7 for Ubuntu1804 r=firefox-build-system-reviewers,glandium
https://hg.mozilla.org/integration/autoland/rev/abf785b6ffe4
Bump Mach's minimum Python version to 3.7 r=firefox-build-system-reviewers,glandium

Comment 9

[Serban Stanca [:SerbanS]]

Backed out for causing mda failures on Android.

• Backout link
• Push with failures
• Failure Log
• Failure line: Could not install python package: failed all attempts.

Comment 10

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

This should work now, as of bug 1804514.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2ad732e17790
Upgrade to Python 3.7 for Ubuntu1804 r=firefox-build-system-reviewers,glandium
https://hg.mozilla.org/integration/autoland/rev/9eaec62e8473
Bump Mach's minimum Python version to 3.7 r=firefox-build-system-reviewers,glandium

Comment 12

[Marian-Vasile Laza]

Backed out 4 changesets (Bug 1734402, Bug 1732228) for build bustages when upgrade pip-23.0.1-py3-none-any.whl.
Backout link
Push with failures
Failure Log

Comment 13

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

That was weird. The .whl files were 0 bytes on Phabricator. I've re-uploaded the patches now and they have the correct file size. I'll reattempt to land this.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2b233ff8eee7
Upgrade to Python 3.7 for Ubuntu1804 r=firefox-build-system-reviewers,glandium
https://hg.mozilla.org/integration/autoland/rev/dbab8862bb8a
Bump Mach's minimum Python version to 3.7 r=firefox-build-system-reviewers,glandium

Comment 15

[Marian-Vasile Laza]

Backed out 4 changesets (Bug 1732228, Bug 1734402) for multiple browsertime and talos failures.
Backout link
Push with failures <--> cnn <--> damp-inspector
cnn Failure Log
damp-inspector Failure Log

Comment 16

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1734402 - Version bumps to `pip`, `setuptools`, and `wheel` to maintain consistency with the wheels used by mozharness r?#build

See D171176

pip==23.0.1
setuptools==67.4.0
wheel==0.38.4

(wheel is not used by mozharness, but we'll just update it here for good measure)

Depends on D171628

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fa1b65b06266
Upgrade to Python 3.7 for Ubuntu1804 r=firefox-build-system-reviewers,glandium
https://hg.mozilla.org/integration/autoland/rev/7c4f9b1462b2
Bump Mach's minimum Python version to 3.7 r=firefox-build-system-reviewers,glandium

Comment 18

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/fa1b65b06266
https://hg.mozilla.org/mozilla-central/rev/7c4f9b1462b2

Comment 19

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1734402 - Regenerate lockfile with Python 3.7 as the minimum version r?#build

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/39ff773cfa63
Regenerate lockfile with Python 3.7 as the minimum version r=ahal

Comment 21

[Cristian Tuns]

Backed out for causing build bustages Python related

• Backout link
• Push with failures
• Failure Log
• Failure line: Exception: According to "pip check", the current Python environment has package-compatibility issues.
  [taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory '/var/opt/generic-worker/tasks/task_167815377605851/logs'", reason "file-missing-on-worker" and expiry 2024-03-06T04:25:24.001Z

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bc631037df77
Regenerate lockfile with Python 3.7 as the minimum version r=ahal

Comment 23

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/bc631037df77

Comment 24

[Phabricator Automation]

Comment on attachment 9321148 [details]
Bug 1734402 - Version bumps to pip, setuptools, and wheel to maintain consistency with the wheels used by mozharness r?#build

Revision D171629 was moved to bug 1732228. Setting attachment 9321148 [details] to obsolete.

Comment 25

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1734402 - Regenerate the `third_party/python` lockfile with Python `3.8` as the minimum version r?#build

• If this is not done, the next person to vendor will also make all of
  the changes in this commit, which is unnecessary noise.

Depends on D183449

Comment 26

[Phabricator Automation]

Comment on attachment 9343621 [details]
Bug 1734402 - Regenerate the third_party/python lockfile with Python 3.8 as the minimum version r?#build

Revision D183450 was moved to bug 1843209. Setting attachment 9343621 [details] to obsolete.