Bug 1388447 (buildpython3)

[meta] Support and Require Python 3 to build Firefox

NEW
Unassigned

Status

2 years ago
19 days ago

People

(Reporter: gps, Unassigned)

Tracking

(Depends on: 7 bugs, Blocks: 2 bugs, {meta})

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox57 wontfix)

Details

(Reporter)

Description

2 years ago
Python 2 will not be maintained past 2020. Python 3 is the future.

Today, we're ready to allow the use of Python 3 for *optional* Firefox development activities. The latest MozillaBuild ships Python 3.6 and Python 3 is generally available on most *NIX distributions.

At some point, we'll require Python 3 to build Firefox. I'm not yet sure when this will be. I'll file another bug to track that.

Eventually, we'll port all in-tree Python to Python 3 and drop support for Python 2. This will likely take years and extend past 2020 (because there aren't great incentives for rewriting working code that isn't actively interfering with other development activities).

This meta bug tracks everything Python 3 as it relates to to developing and building Firefox.
(Reporter)

Updated

2 years ago
Depends on: 1388452
Many things in the mozbuild code would be a major PITA to write as python 2 *and* 3 compatible, most notably things revolving around strings. How do you plan to handle that transition?
(Reporter)

Comment 2

2 years ago
I'm not yet sure how all the code will be transitioned. I imagine Python 3 in the build system will start with some well-isolated component switching to it. Python 3 support will then spread as needed.

Some components will be able to make a clean break from Python 2. e.g. we could potentially flag day things like the xpcshell harness, which don't have too many non-stdlib dependencies. `mach` may then have to spawn a new process instead of importing the test harness in process. So be it. Given how much code is imported by things like `mach`, flag day transitions for many component will be logistically challenging. This is why I suspect most code will need to be 2/3 dual compatible for a while.

I'm not opposed to using custom module importers to pave over differences. The hack I wrote for Mercurial (https://gregoryszorc.com/blog/2017/03/13/from-__past__-import-bytes_literals/) could be employed. We could also use something like py-backwards (https://github.com/nvbn/py-backwards) to go the other way.

I also suspect that we'll need to move towards proper virtualenvs with actual file copies/symlinks instead of sys.path and .pth hacks (which we use aggressively for mach and the build system's virtualenv). This work is years overdue. The direction I'd like to take mach is brain dumped in bug 985141. But there are performance issues with a proper virtualenv. Namely, you need to make sure the virtualenv's files don't need refreshed. This means something resembling a build system to manage the virtualenv. In theory, that is pip or the virtualenv module itself. However, my experience shows that these are way too slow performing the "is up to date" checks. We can't have every `mach` command taking hundreds of milliseconds checking if the virtualenv is current before it does any meaningful work, for example. I'm not even sure if pip adequately captures dependencies so that we know which files to stat() to see if packages are up to date (I think pip's model is basically "run install"). Moving away from sys.path and .pth hacks is going to be painful. So if we can stick to source-level compatibility without requiring an out-of-band or expensive source translation process, that would be good.

FWIW, problems like Python 3 compatibility and responsiveness that result from "import bloat" are why there is a best practice to avoid anything more than dispatch code in mach_commands.py files (https://gecko.readthedocs.io/en/latest/python/mach/commands.html#minimizing-code-in-commands). However, that best practice isn't well enforced (I'm as guilty as anyone). As they say, the chickens have come home to roost.
(Reporter)

Comment 3

2 years ago
I may have been overly pessimistic about pip's performance. On the virtualenv in version-control-tools, a no-op `pip install -r <path>` with ~90 packages takes ~500ms. Another virtualenv with ~10 packages takes ~250ms. I thought it actually took seconds.

We're still about a magnitude slower than we need to be to put this in the critical path of `mach` commands. But we might be close enough that we can hack together something using pip's internals to get the performance we need.
Presuming the packages in the requirements files are all pinned to exact versions and not ranges (which is best practice anyway), then could mach just compare a hash of the files to the hash the last time bootstrap was invoked instead?
To clarify: "hash of the files" -> "hash of the requirements files"
(Reporter)

Updated

2 years ago
Depends on: 636155
(Reporter)

Comment 6

2 years ago
Let's take the discussion of virtualenvs elsewhere. For lack of a better place, how about bug 985141?
(Reporter)

Updated

2 years ago
Depends on: 1388894
(Reporter)

Updated

2 years ago
Depends on: 1391019
(Reporter)

Updated

2 years ago
status-firefox57: --- → wontfix
(Reporter)

Updated

a year ago
Depends on: 1413630
(Reporter)

Comment 7

a year ago
At dinner today, a group of 4 build peers decided that we would like to make Python 3 a build requirement in Firefox 59.

This is because Firefox 59 will be an ESR release and the move towards Python 3 will likely gain a lot of momentum before the ESR after 59. Once we start porting code to Python 3, it may become difficult to backport Python 3 code to work on older Python 2 code. Given the relatively long lifetime of ESR releases, we feel it is best to minimize the potential backporting pain by establishing the Python 3 requirement in ESR 59.

Also, it is quite possible we don't actively use Python 3 in the build system in Firefox 59. But once we establish the requirement in configure checks, we'll be free to do so at any time. That's the point.

We'll likely target Python 3.5+. Although I would love to bump that to 3.6+ just so we can get the latest/greatest Python. The distro support may not be there though. I need to take an inventory of the distros and make an announcement on mailing lists to make sure whatever we do doesn't cause too much chaos.

Setting needinfo me to track that.

Please comment here if you have any meaningful comments about this timeline.
Flags: needinfo?(gps)
(Reporter)

Comment 8

a year ago
Python 3.5 is what ships in Debian 9, Ubuntu 16.04, Mint 18, Fedora 24, Arch >~2015-09 (unsure of Arch's release process).

Notably missing support for Python 3 is the Redhat distributions. RHEL 7 and CentOS 7 don't offer Python 3 packages out of the box. But, it is trivial to add the IUS repo and get the latest version of Python from there. We also have precedent for requiring a Python not available out-of-the box on these distros: we required Python 2.7 before RHEL 7 and CentOS 7 shipped (RHEL/CentOS 6 only had Python 2.6).

I'm not sure what version it first shipped in, but FreeBSD's ports first added 3.5 in September 2015.

As much as I want to target 3.6+, given the current landscape of distro support, that seems difficult. But if the packaging environment of these distros supports 3.6, I'd love to require it.
(Reporter)

Comment 9

a year ago
Will send out email soon.

FWIW, https://docs.python.org/3.6/whatsnew/3.6.html has a list of new features in 3.6. Relevant to Firefox's needs:

* asyncio is stabilized and has a handful of compelling features over 3.5
* support for paths longer than MAX_PATH on Windows
* improvements to debugging primitives that make developer's lives much easier
* tons of performance improvements
Flags: needinfo?(gps)
+1

I just ran into lack of Python 3 support the other day when I tried to adapt .ycm_extra_conf.py for another tool, and it failed because my editor was using Python 3.

I don't have any opinion on dropping Python 2 support. It's been years since I tried to build on a system without Python 3.
> * asyncio is stabilized and has a handful of compelling features over 3.5

It's well worth using 3.6 for smooth asyncio.  I'd hate to have crufty asyncio code in the tree for ESR reasons; it's just a terrible situation.  And asyncio is _the_ reason to use Python 3 IMO; everything else is just wart-shaving.  (I'd consider bytes as compelling, too.)
(In reply to Gregory Szorc [:gps] from comment #7)
> At dinner today, a group of 4 build peers decided that we would like to make
> Python 3 a build requirement in Firefox 59.
> 
> This is because Firefox 59 will be an ESR release and the move towards
> Python 3 will likely gain a lot of momentum before the ESR after 59. Once we
> start porting code to Python 3, it may become difficult to backport Python 3
> code to work on older Python 2 code. Given the relatively long lifetime of
> ESR releases, we feel it is best to minimize the potential backporting pain
> by establishing the Python 3 requirement in ESR 59.

If anything, ESR would be a good reason *not* to require python 3 before *60*. Because saying "python 3.5 or 3.6" is required to build ESR is equivalent to "you won't have security updates for Firefox anymore in a few months" for some distros.

As a data point, Debian stable has python 3.5, oldstable 3.4, oldoldstable 3.2. They currently all ship Firefox ESR52. Oldstable is supported until May next year, stable until June 2020.
Ubuntu 14.04LTS has 3.4, 16.04LTS has 3.5. They both ship current Firefox release (56), and are supported, respectively, until April 2019 and April 2021.
I didn't look at other distros, but they likely have the same problem.
Sure, there's the "rust problem", but it's actually easier to deal with adding it than upgrading python on those "old" stable distros.
(In reply to Mike Hommey [:glandium] from comment #12)
> As a data point, Debian stable has python 3.5, oldstable 3.4, oldoldstable
> 3.2. They currently all ship Firefox ESR52. Oldstable is supported until May
> next year, stable until June 2020.

Err, I shifted things, oldoldstable is supported until May 2018, oldstable until June 2020, stable until June 2022.
Ubuntu 16.04 can have Python 3.6 PPAs installed (just FYI, PPAs may not be desirable)
Ubuntu 17.10 has Python 3.6.3
Fedora 26 has Python 3.6.2
OpenSUSE Leap 42 has Python 3.4.6
Homebrew (macOS X 10.13, for example) has Python 3.6.3
:Waldo pointed this site which may be useful for all distros:

https://repology.org/metapackage/python3/versions
(Reporter)

Comment 16

a year ago
I was convinced by glandium and others to hold off until post ESR before instituting this requirement. I don't think the concern about backporting Python 3 to a Python 2 ESR 60 will be a problem in reality. So we can wait until 61.

Updated

a year ago
Product: Core → Firefox Build System
(Reporter)

Comment 17

a year ago
Nightly is now Firefox 61. So this bug is theoretically unblocked.

Most of the work is likely getting Python 3 installed throughout CI.

I believe we decided our minimum version should be 3.5.
(Reporter)

Comment 18

a year ago
The set of CI platforms where we need to install Python 3 in PATH is apparently "all of them:" https://treeherder.mozilla.org/#/jobs?repo=try&revision=c2ba2dac75b0a69e6b45196945c941b973cfe20f
(Reporter)

Updated

a year ago
Depends on: 1449629
(Reporter)

Updated

a year ago
Depends on: 1451106
(Reporter)

Updated

a year ago
Depends on: 1451107

Comment 19

11 months ago
I'm curious whether or not we intend to allow GENERATED_FILES scripts to be written in Python 3 in the near future. I (think) currently they'll not work nicely as Python 3 scripts as the script which loads them is Python 2.
Your comment is the first I've heard of anyone wanting to do that, so I don't think we've planned anything around it! Most of the work has been to allow individual mach commands to use Python 3. Once we require Python 3 to build then I don't think it'd be a lot of work to allow individual `GENERATED_FILES` scripts to opt-in to using Python 3. We'd basically just have to make sure `file_generate` works in both Python 2 and 3:
https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/file_generate.py

and add some moz.build machinery to opt-in and propagate that down through the build backend.
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #20)
> Your comment is the first I've heard of anyone wanting to do that, so I
> don't think we've planned anything around it! Most of the work has been to
> allow individual mach commands to use Python 3. Once we require Python 3 to
> build then I don't think it'd be a lot of work to allow individual
> `GENERATED_FILES` scripts to opt-in to using Python 3. We'd basically just
> have to make sure `file_generate` works in both Python 2 and 3:
> https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/
> action/file_generate.py
> 
> and add some moz.build machinery to opt-in and propagate that down through
> the build backend.

I have thought about this, and agree with Ted that the machinery is not too difficult.  The challenge for the build system is that lots of things in python/mozbuild aren't (even close) to Python 3 compatible, so it's pretty hard to write non-trivial GENERATED_FILES actions that are 2/3 compatible.
(Reporter)

Updated

11 months ago
Depends on: 1451065
(Reporter)

Updated

11 months ago
Depends on: 1455708
(Reporter)

Updated

10 months ago
Depends on: 1466535
Depends on: 1471620, 1093212
Depends on: 1472177
Depends on: 1473063

Updated

6 months ago
Blocks: 1496527

Updated

5 months ago
Depends on: 1210157

Updated

5 months ago
Depends on: 1487490

Updated

4 months ago
Blocks: 1507655
Depends on: 1507488
You need to log in before you can comment on or make changes to this bug.