Closed Bug 1577599 (mach-py3) Opened 5 years ago Closed 4 years ago

[meta] mach command migration to python3

Tracking

(Not tracked)

Status:

RESOLVED FIXED

People

(Reporter: egao, Unassigned)

References

(Blocks 2 open bugs)

Details

(Keywords: meta)

User Story

** I got a bug filed against me that is blocking this meta bug. What do I do?

The goal here is to run every `mach` command with Python 3. Put another way, we'd like to make sure you can always run any `mach` command on a system that does not have Python 2 installed. Usually, you can close your bug when the `mach` command mentioned in the title is removed from the list of `py2commands` [here](https://searchfox.org/mozilla-central/source/mach#13).

** How do I begin working on this?

The easiest place to start is at the "end"; i.e., remove your command from the `py2commands` list and try running a basic invocation of your `mach` command (which will now be using Python 3). It will probably fail; look at the exception, figure out what's causing it, and fix it, then continue in this fashion until the invocation succeeds.

Python being a dynamic language, branch coverage is especially important (you don't want to submit your patch just to find out that it breaks one branch that you didn't happen to test). If your command can run in multiple "modes", make sure you exercise all those "modes" while testing. Subcommands are an obvious example: if your command supports subcommands, just make sure you test each of them out before sending a patch, because removing the top-level command from the list of `py2commands` will affect every subcommand.

** How do you convert code from Python 2 to Python 3?

This is somewhat out of scope for this short intro, but in general you should learn the differences between Python 2 and Python 3 and be aware of them as you port. With an appropriate amount of expertise, you should be able to look at most exceptions that would arise as a result of doing this and trace it to a well-known difference between Python 2 and 3. The Python project recommends some resources (this [cheat sheet](http://python-future.org/compatible_idioms.html) and [this book](http://python3porting.com/)).

Having said all that, I have two suggestions for you:

1. Try not to get hung up on reading EVERYTHING before just starting. The list of backward-compatibility breakages between Python 2 and 3 is huge and goes way beyond how `print()` statements are formatted, and learning them all ahead of time will very likely not get you anywhere. You'll pretty easily be able to narrow down any exception to some vague problem space (i.e., strings, file I/O, subprocessing, data structures, module renames), and smart Googling will probably get you to the finish line to fix any particular issue.

2. I would recommend not wasting a great deal of time trying to get any value out of auto-converters like Futurize or Modernize. In my experience these tools don't know enough about how your code works to be able to make significant, meaningful changes, so the only changes they actually can confidently make are simple text replacement that you could very easily do yourself. (That's not to say *don't* try using these, but don't expect them to change your entire world for you, and don't expect them to save you any significant amount of time.)

If all else fails, a well-placed question (with enough accompanying context so that people know what's going on, plus a stacktrace if you have one) to #Python on Matrix will probably get you an answer quickly enough.

** How do I finish this?

As mentioned above, you can usually consider your work done when you remove your command from the list of `py2commands`.

Many `mach` commands consist of a small enough amount of code that a single, small, atomic patch can be written to port everything all at once. If that's not the case, then you may find yourself in a situation where the conversion needs to be done over time with a few or more patches. In that case you'll have to maintain your code in a state where it continues to work under Python 2 until you're ready to make the final switchover. (Judicious use of the `six` module, plus the cheat sheet above, and making all the code to be ported already has the boilerplate `from __future__ import absolute_import, print_function, unicode_literals` at the top of the file will be a big help to you here.)

If your `mach` command runs code that is exercised by unit tests, also make sure those unit tests are running Python 3 so regressions don't occur.

You may make progress on the port only to find that you're blocked on something, like a library owned by some other (Mozilla or non-Mozilla) party. If it's a vendored third-party library, you can use `mach vendor python` to pull in a new, Python 3-compatible library. If it's an in-tree library that you depend on but don't own (and you're out of your depth trying to fix the incompatibilities yourself), then as usual, you can file a bug against the owners of that library letting them know about the dependency and that their code doesn't work with Python 3.

** I don't think I'm responsible for doing this port.

Bugs were assigned to Bugzilla components automatically based on source metadata as found in `moz.build` files, so there is room for error in how assignments were done. Finding a more appropriate Bugzilla component, or a particular person, and assigning the bug to them may be in order.

Edwin Takahashi (:egao | infrequent contributor)

Reporter

Description

•

5 years ago

Tracking bug to collect all mach commands being migrated to python3.

Edwin Takahashi (:egao | infrequent contributor)

Reporter