636155 - Switch to requiring Python 3 at some point

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(not currently active) Ted Mielczarek]

This is intentionally vague at the moment. I'm not sure how much work it will be to convert all our in-tree scripts to using Python 3. Python 2.7 is the last Python 2.x release, and all active development is on the 3.x branch. Python 3 is available in most Linux distros at this point, so I don't think it will be a big issue. Supporting both Python 2.x and 3.x is a non-starter AFAICT, but we can probably make our lives a bit easier by enabling the python3k warnings and fixing all of those as a first pass. Then, we'll just have to declare a flag day and switch, probably by running 2to3 on all our scripts and manually fixing the results.

Some other things that should get filed to block this:
* Shipping a Python 3 with MozillaBuild (alongside the existing Python 2).
* Getting Python 3 installed on build slaves (also alongside Python 2).

Comment 1

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Python 3.2 just got released, the 3.2.x branch will probably be a nice candidate to shift to, in the future.

http://www.python.org/download/releases/3.2/

Presumably most distros will come with this version in the future.

CC'ing myk due to bug 613335 (Add-on SDK does not support python 3 or 3.1)

Comment 2

[(not currently active) Ted Mielczarek]

Dunno why you think this blocks bug 578584, it's not really related. Also, when I say "Python 3" here, you can take that to mean "whatever 3.x is current when we get around to this".

Comment 3

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

I couldn't think of a better way to do this at 3am local time but here's what I did by passing in all python scripts on our repo to 2to3.py:

for file in $( ls -d -1 $PWD/*.py ); do python -u /c/Python32/Tools/Scripts/2to3.py $file; done 2>&1 | tee ~/Desktop/logDoubleStar0.txt
for file in $( ls -d -1 $PWD/**/*.py ); do python -u /c/Python32/Tools/Scripts/2to3.py $file; done 2>&1 | tee ~/Desktop/logDoubleStar1.txt
for file in $( ls -d -1 $PWD/**/**/*.py ); do python -u /c/Python32/Tools/Scripts/2to3.py $file; done 2>&1 | tee ~/Desktop/logDoubleStar2.txt
for file in $( ls -d -1 $PWD/**/**/**/*.py ); do python -u /c/Python32/Tools/Scripts/2to3.py $file; done 2>&1 | tee ~/Desktop/logDoubleStar3.txt
for file in $( ls -d -1 $PWD/**/**/**/**/*.py ); do python -u /c/Python32/Tools/Scripts/2to3.py $file; done 2>&1 | tee ~/Desktop/logDoubleStar4.txt
for file in $( ls -d -1 $PWD/**/**/**/**/**/*.py ); do python -u /c/Python32/Tools/Scripts/2to3.py $file; done 2>&1 | tee ~/Desktop/logDoubleStar5.txt
for file in $( ls -d -1 $PWD/**/**/**/**/**/**/*.py ); do python -u /c/Python32/Tools/Scripts/2to3.py $file; done 2>&1 | tee ~/Desktop/logDoubleStar6.txt

I then did "ls -d -1 $PWD/**/**/**/**/**/**/**/*.py" and onwards for 4 more subdirectory levels inclusive just to make sure there are no more:

ls: /c/Users/Zijin/tracemonkey/**/**/**/**/**/**/**/*.py: No such file or directory

(I'm assuming there's no more .py files hidden in any other deeper subdirectories)

Tested this on TM repo 725b8cce5c72, and this turned up bug 636202, which is awaiting approval2.0 at this point in time.

If we pass in the "-w" parameter to 2to3, it will even overwrite the original files for us, making the switch pretty straightforward.

If we want to maintain compatibility with at least Python 2.6 (which is what we ship in MozillaBuild 1.5.1), we could add "from __future__ import print_function" but I'm not sure what more needs to be done, but then again it's already specified as a non-starter in comment #0.

Comment 4

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to comment #2)
> Dunno why you think this blocks bug 578584, it's not really related.

According to http://docs.python.org/py3k/howto/pyporting.html#support-python-2-7 Python 2.7 has the "-3" flag to help "discover places in code which 2to3 cannot handle but are known to cause issues", but I'd suppose it may not be strictly necessary..

Comment 5

[(not currently active) Ted Mielczarek]

Yeah, we don't need Python 2.7 in MozillaBuild to make that happen.

I think you probably want "find . -name "*.py" -exec .../2to3.py {} \;"

Comment 6

[(not currently active) Ted Mielczarek]

From discussion around bug 724191, I think getting everything to Python 2.7 is a good first step. If we have everything on at leas 2.6, then we can import most of the incompatible 3.0 behavior from __future__:
http://docs.python.org/library/__future__.html

Importing division, print_function and unicode_literals would get us pretty far to being 3.0 forwards-compat.

Comment 7

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

We went to requiring Python 2.6 to build the tree, and now we've required Python 2.7 to build the tree for just over a week now.

Comment 9

[Gregory Szorc [:gps]]

Pasting what I wrote recently in bug 1388452:

Python 2 will not be maintained starting sometime in 2020. Python 3 is the future.

At some point, we'll want to require Python 3 to build Firefox. This bug tracks that requirement.

I'm not sure exactly when we'll cross this bridge. We could probably do it soon, as MozillaBuild is now shipping Python 3.6 and I think most *NIX distros have Python easily available as a package. But until we have a compelling reason to require Python 3 to build, I'm not inclined to do it. If nothing else, our adoption of Rust is causing some grief for downstream packagers and I'd prefer to give them a reprieve before we hit them with Python 3.

As for what version of Python 3 to require when we establish the requirement, we should try to require the latest version as possible, as working around bugs and missing features from older versions is a time sink and slows us down. The main impediment to requiring the latest version is difficulties installing that version on various operating system and downstream packaging requirements.

If we made the transition today, I'd strongly prefer to require at least 3.5. The reason is that it is considerably more difficult to write Python that runs on both 2.7 and 3 for versions <3.5. This is due to the lack of the % operator on bytes types, which was restored in 3.5.

If I had to venture a guess, I'd say we'll likely require Python 3 to build sometime in the first half of 2018. But if we find a compelling reason to require it sooner, it could happen in 2017.

Comment 10

[Mike Hommey [:glandium]]

> Python 2 will not be maintained starting sometime in 2020.

Note: that remains to be seen. Not that we shouldn't move towards python 3, but 2020 is probably not a hard deadline. a) it wouldn't be the first time its EOL is extended, b) even if the PSF stops supporting it for real, I'm sure people will take over, even if they have to rename it.

Comment 11

[Gregory Szorc [:gps]]

I agree there will likely be some form of support past 2020, even if it isn't the PSF. RedHat will surely maintain patches for security fixes for years until RHEL's support lifetime ends. I'm sure other vendors will do something similar.

I'd still prefer to have as little exposure to whatever happens though.

Realistically, yes, we'll still be requiring Python 2 to build post 2020. We still have Perl scripts that have been sitting idle for 10+ years!

But I hope we can be mostly Python 3 by then. The code would be faster and easier to hack on, which should enable a faster development velocity. The main loss is the time stolen from other projects we could be working on. Classic technical debt dilemma.

Comment 12

[Gregory Szorc [:gps]]

We required Python 3 to build via a combination of bug 1451065 and bug 1455708. Bug 1388447 seems to be our general tracking bug for "Python 3 all of the things."

I don't see much value keeping this bug open.