1388447 - (buildpython3) [meta] Support and Require Python 3 to build Firefox

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Gregory Szorc [:gps]]

Python 2 will not be maintained past 2020. Python 3 is the future.

Today, we're ready to allow the use of Python 3 for *optional* Firefox development activities. The latest MozillaBuild ships Python 3.6 and Python 3 is generally available on most *NIX distributions.

At some point, we'll require Python 3 to build Firefox. I'm not yet sure when this will be. I'll file another bug to track that.

Eventually, we'll port all in-tree Python to Python 3 and drop support for Python 2. This will likely take years and extend past 2020 (because there aren't great incentives for rewriting working code that isn't actively interfering with other development activities).

This meta bug tracks everything Python 3 as it relates to to developing and building Firefox.

Comment 1

[Mike Hommey [:glandium]]

Many things in the mozbuild code would be a major PITA to write as python 2 *and* 3 compatible, most notably things revolving around strings. How do you plan to handle that transition?

Comment 2

[Gregory Szorc [:gps]]

I'm not yet sure how all the code will be transitioned. I imagine Python 3 in the build system will start with some well-isolated component switching to it. Python 3 support will then spread as needed.

Some components will be able to make a clean break from Python 2. e.g. we could potentially flag day things like the xpcshell harness, which don't have too many non-stdlib dependencies. `mach` may then have to spawn a new process instead of importing the test harness in process. So be it. Given how much code is imported by things like `mach`, flag day transitions for many component will be logistically challenging. This is why I suspect most code will need to be 2/3 dual compatible for a while.

I'm not opposed to using custom module importers to pave over differences. The hack I wrote for Mercurial (https://gregoryszorc.com/blog/2017/03/13/from-__past__-import-bytes_literals/) could be employed. We could also use something like py-backwards (https://github.com/nvbn/py-backwards) to go the other way.

I also suspect that we'll need to move towards proper virtualenvs with actual file copies/symlinks instead of sys.path and .pth hacks (which we use aggressively for mach and the build system's virtualenv). This work is years overdue. The direction I'd like to take mach is brain dumped in bug 985141. But there are performance issues with a proper virtualenv. Namely, you need to make sure the virtualenv's files don't need refreshed. This means something resembling a build system to manage the virtualenv. In theory, that is pip or the virtualenv module itself. However, my experience shows that these are way too slow performing the "is up to date" checks. We can't have every `mach` command taking hundreds of milliseconds checking if the virtualenv is current before it does any meaningful work, for example. I'm not even sure if pip adequately captures dependencies so that we know which files to stat() to see if packages are up to date (I think pip's model is basically "run install"). Moving away from sys.path and .pth hacks is going to be painful. So if we can stick to source-level compatibility without requiring an out-of-band or expensive source translation process, that would be good.

FWIW, problems like Python 3 compatibility and responsiveness that result from "import bloat" are why there is a best practice to avoid anything more than dispatch code in mach_commands.py files (https://gecko.readthedocs.io/en/latest/python/mach/commands.html#minimizing-code-in-commands). However, that best practice isn't well enforced (I'm as guilty as anyone). As they say, the chickens have come home to roost.

Comment 3

[Gregory Szorc [:gps]]

I may have been overly pessimistic about pip's performance. On the virtualenv in version-control-tools, a no-op `pip install -r <path>` with ~90 packages takes ~500ms. Another virtualenv with ~10 packages takes ~250ms. I thought it actually took seconds.

We're still about a magnitude slower than we need to be to put this in the critical path of `mach` commands. But we might be close enough that we can hack together something using pip's internals to get the performance we need.

Comment 4

[Ed Morley [:emorley]]

Presuming the packages in the requirements files are all pinned to exact versions and not ranges (which is best practice anyway), then could mach just compare a hash of the files to the hash the last time bootstrap was invoked instead?

Comment 5

[Ed Morley [:emorley]]

To clarify: "hash of the files" -> "hash of the requirements files"

Comment 6

[Gregory Szorc [:gps]]

Let's take the discussion of virtualenvs elsewhere. For lack of a better place, how about bug 985141?

Comment 7

[Gregory Szorc [:gps]]

At dinner today, a group of 4 build peers decided that we would like to make Python 3 a build requirement in Firefox 59.

This is because Firefox 59 will be an ESR release and the move towards Python 3 will likely gain a lot of momentum before the ESR after 59. Once we start porting code to Python 3, it may become difficult to backport Python 3 code to work on older Python 2 code. Given the relatively long lifetime of ESR releases, we feel it is best to minimize the potential backporting pain by establishing the Python 3 requirement in ESR 59.

Also, it is quite possible we don't actively use Python 3 in the build system in Firefox 59. But once we establish the requirement in configure checks, we'll be free to do so at any time. That's the point.

We'll likely target Python 3.5+. Although I would love to bump that to 3.6+ just so we can get the latest/greatest Python. The distro support may not be there though. I need to take an inventory of the distros and make an announcement on mailing lists to make sure whatever we do doesn't cause too much chaos.

Setting needinfo me to track that.

Please comment here if you have any meaningful comments about this timeline.

Comment 8

[Gregory Szorc [:gps]]

Python 3.5 is what ships in Debian 9, Ubuntu 16.04, Mint 18, Fedora 24, Arch >~2015-09 (unsure of Arch's release process).

Notably missing support for Python 3 is the Redhat distributions. RHEL 7 and CentOS 7 don't offer Python 3 packages out of the box. But, it is trivial to add the IUS repo and get the latest version of Python from there. We also have precedent for requiring a Python not available out-of-the box on these distros: we required Python 2.7 before RHEL 7 and CentOS 7 shipped (RHEL/CentOS 6 only had Python 2.6).

I'm not sure what version it first shipped in, but FreeBSD's ports first added 3.5 in September 2015.

As much as I want to target 3.6+, given the current landscape of distro support, that seems difficult. But if the packaging environment of these distros supports 3.6, I'd love to require it.

Comment 9

[Gregory Szorc [:gps]]

Will send out email soon.

FWIW, https://docs.python.org/3.6/whatsnew/3.6.html has a list of new features in 3.6. Relevant to Firefox's needs:

* asyncio is stabilized and has a handful of compelling features over 3.5
* support for paths longer than MAX_PATH on Windows
* improvements to debugging primitives that make developer's lives much easier
* tons of performance improvements

Comment 10

[Kris Maglione [:kmag]]

+1

I just ran into lack of Python 3 support the other day when I tried to adapt .ycm_extra_conf.py for another tool, and it failed because my editor was using Python 3.

I don't have any opinion on dropping Python 2 support. It's been years since I tried to build on a system without Python 3.

Comment 11

[Nick Alexander :nalexander [he/him]]

> * asyncio is stabilized and has a handful of compelling features over 3.5

It's well worth using 3.6 for smooth asyncio.  I'd hate to have crufty asyncio code in the tree for ESR reasons; it's just a terrible situation.  And asyncio is _the_ reason to use Python 3 IMO; everything else is just wart-shaving.  (I'd consider bytes as compelling, too.)

Comment 12

[Mike Hommey [:glandium]]

(In reply to Gregory Szorc [:gps] from comment #7)
> At dinner today, a group of 4 build peers decided that we would like to make
> Python 3 a build requirement in Firefox 59.
> 
> This is because Firefox 59 will be an ESR release and the move towards
> Python 3 will likely gain a lot of momentum before the ESR after 59. Once we
> start porting code to Python 3, it may become difficult to backport Python 3
> code to work on older Python 2 code. Given the relatively long lifetime of
> ESR releases, we feel it is best to minimize the potential backporting pain
> by establishing the Python 3 requirement in ESR 59.

If anything, ESR would be a good reason *not* to require python 3 before *60*. Because saying "python 3.5 or 3.6" is required to build ESR is equivalent to "you won't have security updates for Firefox anymore in a few months" for some distros.

As a data point, Debian stable has python 3.5, oldstable 3.4, oldoldstable 3.2. They currently all ship Firefox ESR52. Oldstable is supported until May next year, stable until June 2020.
Ubuntu 14.04LTS has 3.4, 16.04LTS has 3.5. They both ship current Firefox release (56), and are supported, respectively, until April 2019 and April 2021.
I didn't look at other distros, but they likely have the same problem.
Sure, there's the "rust problem", but it's actually easier to deal with adding it than upgrading python on those "old" stable distros.

Comment 13

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #12)
> As a data point, Debian stable has python 3.5, oldstable 3.4, oldoldstable
> 3.2. They currently all ship Firefox ESR52. Oldstable is supported until May
> next year, stable until June 2020.

Err, I shifted things, oldoldstable is supported until May 2018, oldstable until June 2020, stable until June 2022.

Comment 14

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Ubuntu 16.04 can have Python 3.6 PPAs installed (just FYI, PPAs may not be desirable)
Ubuntu 17.10 has Python 3.6.3
Fedora 26 has Python 3.6.2
OpenSUSE Leap 42 has Python 3.4.6
Homebrew (macOS X 10.13, for example) has Python 3.6.3

Comment 15

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

:Waldo pointed this site which may be useful for all distros:

https://repology.org/metapackage/python3/versions

Comment 16

[Gregory Szorc [:gps]]

I was convinced by glandium and others to hold off until post ESR before instituting this requirement. I don't think the concern about backporting Python 3 to a Python 2 ESR 60 will be a problem in reality. So we can wait until 61.

Comment 17

[Gregory Szorc [:gps]]

Nightly is now Firefox 61. So this bug is theoretically unblocked.

Most of the work is likely getting Python 3 installed throughout CI.

I believe we decided our minimum version should be 3.5.

Comment 18

[Gregory Szorc [:gps]]

The set of CI platforms where we need to install Python 3 in PATH is apparently "all of them:" https://treeherder.mozilla.org/#/jobs?repo=try&revision=c2ba2dac75b0a69e6b45196945c941b973cfe20f

Comment 19

[Nika Layzell [:nika] (ni? for response)]

I'm curious whether or not we intend to allow GENERATED_FILES scripts to be written in Python 3 in the near future. I (think) currently they'll not work nicely as Python 3 scripts as the script which loads them is Python 2.

Comment 20

[(not currently active) Ted Mielczarek]

Your comment is the first I've heard of anyone wanting to do that, so I don't think we've planned anything around it! Most of the work has been to allow individual mach commands to use Python 3. Once we require Python 3 to build then I don't think it'd be a lot of work to allow individual `GENERATED_FILES` scripts to opt-in to using Python 3. We'd basically just have to make sure `file_generate` works in both Python 2 and 3:
https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/file_generate.py

and add some moz.build machinery to opt-in and propagate that down through the build backend.

Comment 21

[Nick Alexander :nalexander [he/him]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #20)
> Your comment is the first I've heard of anyone wanting to do that, so I
> don't think we've planned anything around it! Most of the work has been to
> allow individual mach commands to use Python 3. Once we require Python 3 to
> build then I don't think it'd be a lot of work to allow individual
> `GENERATED_FILES` scripts to opt-in to using Python 3. We'd basically just
> have to make sure `file_generate` works in both Python 2 and 3:
> https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/
> action/file_generate.py
> 
> and add some moz.build machinery to opt-in and propagate that down through
> the build backend.

I have thought about this, and agree with Ted that the machinery is not too difficult.  The challenge for the build system is that lots of things in python/mozbuild aren't (even close) to Python 3 compatible, so it's pretty hard to write non-trivial GENERATED_FILES actions that are 2/3 compatible.

Comment 22

[Andrew Halberstadt [:ahal]]

(In reply to Gregory Szorc [:gps] from comment #17)

I believe we decided our minimum version should be 3.5.

This comment was posted over a year ago. Does anyone know if this is still the case? I'd love to target 3.6 as a minimum version. I don't think it's too terrible to ask users of older distros to add a PPA. Or if that isn't acceptable, we could download and setup an installation for them in ~/.mozbuild as part of bootstrap and have mach look there first.

It's also worth noting that machines in CI seem to be using 3.7. We'll want to downgrade them to whatever we decide our minimum is.

Comment 23

[Joel Maher ( :jmaher ) (UTC -8)]

raptor in CI might require 3.7, if we downgrade this could affect automation and tools that rely on python 3 currently.

:rwood, do you know about python 3 requirements for raptor and mitmproxy? (and is this changing for browsertime)?

Comment 24

[Robert Wood [:rwood]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #23)

raptor in CI might require 3.7, if we downgrade this could affect automation and tools that rely on python 3 currently.

:rwood, do you know about python 3 requirements for raptor and mitmproxy? (and is this changing for browsertime)?

I don't know offhand re: Python 3.x for raptor/mitmproxy sorry, :bebe do you know (or can you look into that please), thanks! There are alot of mozharness/mozbase dependencies that Raptor uses.

Nick, do you know off hand re: Python 3 and browsertime? Thanks!

Comment 25

[Andrew Halberstadt [:ahal]]

This bug is probably the wrong place to get into details on specific use cases.

In general, our tools should try to support the same version of Python that the build system does. If they don't they should be modified to add support. If for some reason that's not possible, I'm sure the occasional exception can be made.

Note downgrading in CI doesn't necessarily mean removing 3.7, but adding the lower version and making sure we use that by default in our tasks.

Comment 26

[Florin Strugariu [:Bebe]]

:rwood as you stated earlier Raptor has a some non-python 3 dependency in mozharness and mozbase. Raptor itself is python 3 ready.

For mitmproxy it is build and requires python 3 to run from the python package so no worries there

Comment 27

[Nick Alexander :nalexander [he/him]]

(In reply to Robert Wood [:rwood] from comment #24)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #23)

raptor in CI might require 3.7, if we downgrade this could affect automation and tools that rely on python 3 currently.

:rwood, do you know about python 3 requirements for raptor and mitmproxy? (and is this changing for browsertime)?

I don't know offhand re: Python 3.x for raptor/mitmproxy sorry, :bebe do you know (or can you look into that please), thanks! There are alot of mozharness/mozbase dependencies that Raptor uses.

Nick, do you know off hand re: Python 3 and browsertime? Thanks!

I really don't know much about Python 3 compatibility or requirements. There is essentially no browsertime-specific Python code at this time, and what we add can be made Python 3 compatible quite easily I'm sure (assuming, I guess, minimal support for actually running with Python 3 in CI, which all seems to be moving smoothly).

So if the Raptor harness and the underlying mozbase code is Python 3 ready, I don't think browsertime will get in the way at all.

Comment 28

[Mike Hommey [:glandium]]

The Linux builds on automation are using python 3.5. I think it's fair to maybe require a newer version for tests, but we should at least keep 3.5 for the build system.

I don't think it's too terrible to ask users of older distros to add a PPA. Or if that isn't acceptable, we could download and setup an installation for them in ~/.mozbuild as part of bootstrap and have mach look there first.

Actually, neither is really acceptable downstream (especially the second) ; FWIW, distros with ongoing support do come with python 3.5.

Comment 29

[Clément Lefèvre]

(In reply to Mike Hommey [:glandium] from comment #28)

The Linux builds on automation are using python 3.5. I think it's fair to maybe require a newer version for tests, but we should at least keep 3.5 for the build system.

I don't think it's too terrible to ask users of older distros to add a PPA. Or if that isn't acceptable, we could download and setup an installation for them in ~/.mozbuild as part of bootstrap and have mach look there first.

Actually, neither is really acceptable downstream (especially the second) ; FWIW, distros with ongoing support do come with python 3.5.

Well, it depends of the distro, but for the main server ones in their latest version, it's greater versions:
Ubuntu, 3.6 on LTS, 3.7 on stable (https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=python3&searchon=names)
Debian buster is on 3.7 (https://packages.debian.org/buster/python3), which is now the latest stable since a week
Latest RHEL seems to be 3.6 (https://developers.redhat.com/blog/2018/11/14/python-in-rhel-8/)

Comment 30

[Mike Hommey [:glandium]]

Ubuntu Xenial, still supported until 2021, has python3.5 and doesn't have greater versions.
Debian stretch, still supported until 2020 officially, 2022 LTS, has python3.5, and doesn't have greater versions.

Comment 31

[Clément Lefèvre]

My bad, thought the question was whether greater versions were available or not on those distros.

Comment 32

[Andrew Halberstadt [:ahal]]

(In reply to Mike Hommey [:glandium] from comment #28)

The Linux builds on automation are using python 3.5. I think it's fair to maybe require a newer version for tests, but we should at least keep 3.5 for the build system.

Sounds good. The downside to using a different version for non-build things is that people might get used to using e.g f-strings and accidentally add one to a shared library that the build uses (e.g mozbase). Not a huge deal as it will get backed out, but could still be annoying.

Whether or not we standardize on a single Python version, we'll need to make sure the unittests for various modules are all running with the minimum version they need to support.

(In reply to Mike Hommey [:glandium] from comment #28)

Actually, neither is really acceptable downstream (especially the second) ; FWIW, distros with ongoing support do come with python 3.5.

Ubuntu 14.04 LTS (eol April 2022), seems to come with 3.4. By that logic shouldn't 3.4 be the minimum? Also 16.04 eol is 2024. Will the build system be stuck on 3.5 for another 5 years? If that's the case, then yeah.. maybe it would be better to use a more modern version for non-build things in the meantime.

Comment 33

[Ricky Stewart]

We both 1) support and 2) require Python 3 to build Firefox today. I'm closing this bug as complete in light of that.