1638094 - Consider providing documentation on how to install Python 2 on various systems

Status: RESOLVED WONTFIX

(Developer Infrastructure :: Source Documentation, task)

Description

[Andrew Halberstadt [:ahal]]

We have many commands that still run on Python 2, but package managers are already dropping it from their repositories. For instance, brew dropped it here:
https://github.com/Homebrew/homebrew-core/pull/49796

We were currently relying on brew to install it for us here:
https://searchfox.org/mozilla-central/source/python/mozboot/mozboot/osx.py#350

I would of course love to just get all our mach commands converted ASAP, but apparently developers are already feeling the pain caused by this (I heard reports of days being wasted). So seeing as I think the migration will still take awhile, we should probably add code to bootstrap Python 2 from source and help developers avoid the pain of needing to do it themselves.

Comment 1

[Andrew Halberstadt [:ahal]]

Alternatively we could get a list of the most popular mach commands and start a more concerted effort to burn them down such that only the obscure commands still need Python 2 (and therefore this become less important). I think the testing commands are likely the next most popular ones that haven't been converted yet.

glob, would like to get your thoughts on this. I think between our teams we'd want someone to look into this sooner rather than later, especially as more and more people upgrade to systems that have package managers that have dropped Python 2 already.

Comment 2

[Ricky Stewart]

Bug 1637386 talks about whether bootstrap shouldn't install Python 2 any more, and you can leave a comment there if you have thoughts about that.

Needless to say, I really do not think it's in our best interest to do ad-hoc building, packaging, and distribution of a dead language that isn't getting security updates any more.

Regarding the question about mach command usage, you're right that the next big stumbling block is going to be the test commands. mach test, mochitest, and xpcshell-test are all commands that are in the top 10 most used mach commands (about 20% of invocations of those top 10 mach commands are these 3).

For what it's worth Kris Wright and I have been working on filing a bunch of bugs for the remaining mach commands that haven't been ported to Python 3 yet. Hopefully making the issue more visible in Bugzilla will motivate people to get this done.

Comment 3

[Andrew Halberstadt [:ahal]]

I agree that having a global mach bootstrap for everything is not great (see my proposal in bug 1526021). I also agree that writing code to bootstrap deprecated technology is also not great.

Though the reality is that many developer hours will be wasted here if we don't. In the best case scenario, test harness conversion alone is likely months off. These are commands that are used daily.

Though I think there may be a solution #3:

Write a good documentation page that details how to install Python 2 manually on each platform. Then update the mach driver to print an error message that links to it when a developer attempts to run a command requiring Python 2 when not installed.

Comment 4

[Ricky Stewart]

Yeah, solution #3 seems fine to me. I would sign off on that patch. (Of course, I don't know who's going to write that documentation...)

Comment 5

[Mike Hommey [:glandium]]

See https://stackoverflow.com/questions/60298514/brew-reinstalling-python2 for how to install python 2 with brew even now that they removed it.

Comment 6

[ISHIKAWA, Chiaki]

See https://bugzilla.mozilla.org/show_bug.cgi?id=1630345#c5 for a workaround on a Debian GNU/Linux system that references Debian's testing repository. YMMV.

Comment 7

[:glob ✱]

(In reply to Ricky Stewart from comment #2)

Needless to say, I really do not think it's in our best interest to do ad-hoc building, packaging, and distribution of a dead language that isn't getting security updates any more.

+1

We absolutely should not take on ownership of installing a Python 2 environment.

I'm lukewarm on providing documentation, but as it's a much more viable solution than maintaining our own py2 build env I'll update this bug's summary to reflect that.

Comment 8

[:glob ✱]

(In reply to Andrew Halberstadt [:ahal] from comment #1)

Alternatively we could get a list of the most popular mach commands and start a more concerted effort to burn them down such that only the obscure commands still need Python 2 (and therefore this become less important). I think the testing commands are likely the next most popular ones that haven't been converted yet.
glob, would like to get your thoughts on this. I think between our teams we'd want someone to look into this sooner rather than later, especially as more and more people upgrade to systems that have package managers that have dropped Python 2 already.

This is my preferred approach - I've already started discussions with the EE leads to raise awareness of this issue.

We have telemetry on which commands are run the most - here's the top 10 py2 commands for invocations since the start of this year:

test            51,343
xpcshell-test   43,363
mochitest       38,958
clang-format    34,393
taskgraph        6,783
python           5,939
wpt              5,447
gtest            5,187
file-info        4,825
python-test      3,186

Comment 9

[Greg Tatum [:gregtatum]]

Alternatively we could get a list of the most popular mach commands and start a more concerted effort to burn them down

As a Gecko engineer, this would be a great outcome because I mostly only interact with the top 4 commands listed there, and it would mitigate the recent issues I've been personally running into this week. Thanks for looking into this!

Comment 10

[Andrew Halberstadt [:ahal]]

test            51,343
xpcshell-test   43,363
mochitest       38,958
clang-format    34,393
taskgraph        6,783
python           5,939
wpt              5,447
gtest            5,187
file-info        4,825
python-test      3,186

Note: while |mach test| is the next most common, it will have to be converted last (after all other harnesses it supports are converted). Those other harnesses will also need to maintain Py2 compatibility until |mach test| is converted.

Comment 11

[Ricky Stewart]

Following up on the feedback above, thanks to Kris for filing the remaining Py2 bugs as blockers of bug 1577599. I think every remaining Py2 mach command has a bug now. I also updated the documentation to hopefully make the process as seamless as possible.

Comment 12

[Andrew Halberstadt [:ahal]]

We're almost there, just a few uncommon test commands left.