Closed Bug 1637386 Opened 4 years ago Closed 3 years ago

Bootstrap shouldn't install Python 2 stuff

Categories

(Firefox Build System :: Bootstrap Configuration, task)

Product:
Firefox Build System
Component:
Bootstrap Configuration
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1647814

People

(Reporter: rstewart, Assigned: rstewart)

References

Details

Since last week we don't require Python 2 for builds, so we don't need this stuff in bootstrap any more.

(In reply to Ricky Stewart from comment #0)

Since last week we don't require Python 2 for builds, so we don't need this stuff in bootstrap any more.

We still have some non-trivial number of commands that run with Python 2, though? Should we still have Python 2 for those commands?

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Component: General → Bootstrap Configuration

My thoughts on that:

  1. mach bootstrap is officially for building and nothing else (i.e., mach help bootstrap says the command is to "Install required system packages for building", and the message at the end of bootstrap says your system is "ready to build"), so I don't think we're beholden to anything but the build in principle. There are various mach commands that depend on other, non-build stuff and they handle that on their own without relying on bootstrap to install and update their dependencies.
  2. As far as I can tell, some mach commands are going to depend on Python 2 in the forseeable future. We're continuously working through the low- and medium-hanging fruit but the high-hanging fruit is going to be there for a non-negligible amount of time and we can't be blocked on that.
  3. Python 2 support in OS's is fading fast so looking forward it's unclear what it even means for bootstrap to install Python 2. We're already not doing it on the latest versions of Debian/Ubuntu. I'm really not interested in maintaining bootstrap and keeping up with further OS updates that delete Python 2 infrastructure after its EOL. Patches like this one take (sometimes substantial) engineering time without adding any value for anyone.

I think all of these reasons combined are good enough to move forward with this, but I'm open to discussion and if there's a consensus that we should wait on some external signal then we can postpone for a bit.

  1. mach bootstrap is officially for building and nothing else

It is possible (likely?) that the wording there dates from the original days of a much more limited mach, and hasn't caught up with reality. I suspect that people don't know/care about the fine print and just want to do their work. We should think about how to serve developers best in practice. To be clear, I'm not necessarily against the proposal at this point, I just don't think that reason #1 should be part of the justification.

  1. As far as I can tell, some mach commands are going to depend on Python 2 in the forseeable future.

Can the build system telemetry help here? Can we establish some bar of "commands that X% of developers use Y% of the time don't need Python 2" before proceeding?

I hit save too early. What I wanted to add was, I think it's reasonable if some commands stay on py2 for a long time, if they are esoteric or "you are clearly in self-support territory" type things. If the commands are seeing day-to-day usage that's different. That's where I think seeing a distribution of the usage would be useful.

I suspect that people don't know/care about the fine print and just want to do their work. We should think about how to serve developers best in practice. To be clear, I'm not necessarily against the proposal at this point, I just don't think that reason #1 should be part of the justification.

Yeah, I don't know. This definitely isn't the fine print and is just the... regular print. In general I'm not moved by arguments that we need to hinder ourselves because other people ended up taking implicit dependencies on implementation details that we never actually exposed as part of a contract. I'm open to making exceptions to that rule where pragmatic to avoid breaking wide swaths of people, but this won't break anyone. Anyone who's currently working on Firefox already has Python 2 installed so this strictly affects people who are starting fresh with new dev environments, and they can install Python themselves, it's not hard (unless your OS doesn't allow you to, in which case bootstrap wouldn't have been able to do it for you either, so the point is moot). You only have to do it once as well, since it's not like Python 2 is changing at any point in the future.

All that said, even if we ignore all of that and completely exclude reason #1 as you suggested, I think there's more than enough stuff to talk about here anyway.

Tangential, but if the documentation of bootstrap doesn't match what the actual contract of bootstrap is (or if we're unclear on what the actual contract of bootstrap currently is), then we should fix that.

Can the build system telemetry help here? Can we establish some bar of "commands that X% of developers use Y% of the time don't need Python 2" before proceeding?

*** EDIT: The dashboard already has some version of this info, to an extent. Let me take a look at that.

The dashboard has limited data on mach command invocations but of the top 10 mach commands, 26.3% of invocations (by total count) are still running Python 2. (I think clang-format is low-hanging fruit to port to Python 3 and I'm trying it now. If that goes through then <20% will require Python 2.)

OK, that's in bug 1637664.

The remainder of the commonly used py2 mach commands that aren't low-hanging fruit is all test stuff. I don't know if that team is making any active progress toward py3-ifying their stuff.

Fyi, I just filed bug 1638094 because I heard reports of developers wasting half a day trying to get Python 2 installed manually (they were on a system with a package manager that had already removed it).

See Also: → 1638094

(In reply to Ricky Stewart from comment #3)

"Install required system packages for building", and the message at the end of bootstrap says your system is "ready to build")

I suspect that "build" there is meant expansively. At the time that that was written, I'd bet that the team working on build things also included the test infrastructure and so there was not a lot of thought put into the distinction between build and test. I'll also note that one of the options that mach bootstrap gives you is to setup an artifact build, which I would is very much not build like, and which is designed to make it easy to /test/ changes to the js without needing to do a build.

Bootstrapping is definitely not a unique problem to the build system. But having a single command that bootstraps the world, is a terrible UX. In bug 1526021 I outline a better approach.

Anyway, if it's all the same.. I'd ask if we could hold off on this bug, at least until we have the bulk of the testing commands converted.

The downside of leaving this alone for a few more months seems negligible.

For reference here's the top 10 py2 commands for invocations since the start of this year:

test            51,343
xpcshell-test   43,363
mochitest       38,958
clang-format    34,393
taskgraph        6,783
python           5,939
wpt              5,447
gtest            5,187
file-info        4,825
python-test      3,186

I agree one of the big open questions that comes out of this is "what is bootstrap, and what purpose does it serve" (i.e., exactly which group of people should be able to only run bootstrap and then accomplish their basic day-to-day dev tasks). Things are fuzzier here than I might prefer.

(In reply to Andrew Halberstadt [:ahal] from comment #13)

The downside of leaving this alone for a few more months seems negligible.

Speak for yourself. :P

Bug 1626098 (and the related circle of bootstrap bugs that popped up when the latest pre-release versions of Debian and Ubuntu dropped) was a disaster and if the comments on the bug are any indication, it wasn't even a complete solution to the problem. I'm anticipating the hours of work I'll have to do whenever similar work needs to be done for other operating systems, or whenever the next round of bugs about Debian and Ubuntu come in. And to be clear, none of this work creates any value besides allowing you to not run a separate shell one-liner to install Python 2 if you already were able to do so.

Again, to be clear, maybe the resolution to this bug will be to wait a few months for testing to catch up as proposed above, but I would really prefer that we not pretend that this incurs no ongoing maintenance cost.

mozilla/python/mozboot/mozboot/debian.py seems to have been changed and now |mach bootstrap| works under Debian GNU/Linux amd64.
(So I took out my local patch and let the M-C tree version run, and it works.)
Thank you !
My /etc/debian_version contains "bullseye/sid" and I use testing repository.

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.