Closed Bug 1577599 (mach-py3) Opened 2 years ago Closed 4 months ago

[meta] mach command migration to python3

Categories

(Firefox Build System :: General, enhancement)

enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: egao, Unassigned)

References

(Blocks 2 open bugs)

Details

(Keywords: meta)

User Story

** I got a bug filed against me that is blocking this meta bug. What do I do?

The goal here is to run every `mach` command with Python 3. Put another way, we'd like to make sure you can always run any `mach` command on a system that does not have Python 2 installed. Usually, you can close your bug when the `mach` command mentioned in the title is removed from the list of `py2commands` [here](https://searchfox.org/mozilla-central/source/mach#13).

** How do I begin working on this?

The easiest place to start is at the "end"; i.e., remove your command from the `py2commands` list and try running a basic invocation of your `mach` command (which will now be using Python 3). It will probably fail; look at the exception, figure out what's causing it, and fix it, then continue in this fashion until the invocation succeeds.

Python being a dynamic language, branch coverage is especially important (you don't want to submit your patch just to find out that it breaks one branch that you didn't happen to test). If your command can run in multiple "modes", make sure you exercise all those "modes" while testing. Subcommands are an obvious example: if your command supports subcommands, just make sure you test each of them out before sending a patch, because removing the top-level command from the list of `py2commands` will affect every subcommand.

** How do you convert code from Python 2 to Python 3?

This is somewhat out of scope for this short intro, but in general you should learn the differences between Python 2 and Python 3 and be aware of them as you port. With an appropriate amount of expertise, you should be able to look at most exceptions that would arise as a result of doing this and trace it to a well-known difference between Python 2 and 3. The Python project recommends some resources (this [cheat sheet](http://python-future.org/compatible_idioms.html) and [this book](http://python3porting.com/)). 

Having said all that, I have two suggestions for you:

1. Try not to get hung up on reading EVERYTHING before just starting. The list of backward-compatibility breakages between Python 2 and 3 is huge and goes way beyond how `print()` statements are formatted, and learning them all ahead of time will very likely not get you anywhere. You'll pretty easily be able to narrow down any exception to some vague problem space (i.e., strings, file I/O, subprocessing, data structures, module renames), and smart Googling will probably get you to the finish line to fix any particular issue.

2. I would recommend not wasting a great deal of time trying to get any value out of auto-converters like Futurize or Modernize. In my experience these tools don't know enough about how your code works to be able to make significant, meaningful changes, so the only changes they actually can confidently make are simple text replacement that you could very easily do yourself. (That's not to say *don't* try using these, but don't expect them to change your entire world for you, and don't expect them to save you any significant amount of time.)

If all else fails, a well-placed question (with enough accompanying context so that people know what's going on, plus a stacktrace if you have one) to #Python on Matrix will probably get you an answer quickly enough.

** How do I finish this?

As mentioned above, you can usually consider your work done when you remove your command from the list of `py2commands`.

Many `mach` commands consist of a small enough amount of code that a single, small, atomic patch can be written to port everything all at once. If that's not the case, then you may find yourself in a situation where the conversion needs to be done over time with a few or more patches. In that case you'll have to maintain your code in a state where it continues to work under Python 2 until you're ready to make the final switchover. (Judicious use of the `six` module, plus the cheat sheet above, and making all the code to be ported already has the boilerplate `from __future__ import absolute_import, print_function, unicode_literals` at the top of the file will be a big help to you here.)

If your `mach` command runs code that is exercised by unit tests, also make sure those unit tests are running Python 3 so regressions don't occur.

You may make progress on the port only to find that you're blocked on something, like a library owned by some other (Mozilla or non-Mozilla) party. If it's a vendored third-party library, you can use `mach vendor python` to pull in a new, Python 3-compatible library. If it's an in-tree library that you depend on but don't own (and you're out of your depth trying to fix the incompatibilities yourself), then as usual, you can file a bug against the owners of that library letting them know about the dependency and that their code doesn't work with Python 3.

** I don't think I'm responsible for doing this port. 

Bugs were assigned to Bugzilla components automatically based on source metadata as found in `moz.build` files, so there is room for error in how assignments were done. Finding a more appropriate Bugzilla component, or a particular person, and assigning the bug to them may be in order.

Tracking bug to collect all mach commands being migrated to python3.

Alias: mach-py3
Type: task → defect
Type: defect → enhancement
Blocks: 1606193
No longer blocks: 1606193
Depends on: 1606193
Depends on: 1638947
Depends on: 1638948
Depends on: 1638949
Depends on: 1638950
Depends on: 1638951
Depends on: 1638952
Depends on: 1638953
Depends on: 1638954
Depends on: 1638955
Depends on: 1638956
Depends on: 1638957
Depends on: 1638959
Depends on: 1638960
Depends on: 1638961
Depends on: 1638962
Depends on: 1638963
Depends on: 1638964
Depends on: 1638965
Depends on: 1638966
Depends on: 1638967
Depends on: 1638968
Depends on: 1638970
Depends on: 1638971
Depends on: 1638972
Depends on: 1638973
Depends on: 1638974
Depends on: 1638975
Depends on: 1638976
Depends on: 1638977
Depends on: 1638978
Depends on: 1638979
Depends on: 1638980
Depends on: 1638981
Depends on: 1638982
Depends on: 1638983
Depends on: 1638984
Depends on: 1638985
Depends on: 1638986
Depends on: 1638987
Depends on: 1638989
Depends on: 1638990
Depends on: 1638991
Depends on: 1638992
Depends on: 1638993
Depends on: 1638994
Depends on: 1638995
Depends on: 1638996
Depends on: 1638997
Depends on: 1638998
Depends on: 1638999
Depends on: 1639000
Depends on: 1639001
Depends on: 1639002
Depends on: 1639003
Depends on: 1639004
Depends on: 1639005
Depends on: 1639006
Depends on: 1639007
Depends on: 1639008
Depends on: 1639009
Depends on: 1639024
Depends on: 1639025
Depends on: 1520458
User Story: (updated)
User Story: (updated)
User Story: (updated)
User Story: (updated)
User Story: (updated)
User Story: (updated)
User Story: (updated)
Depends on: 1642313

All mach commands have been converted. Follow-up work to de-support Python 2 will be tracked via the goodbye-py2 meta bug.

Status: NEW → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
See Also: → 1713825
You need to log in before you can comment on or make changes to this bug.