1580422 - mach - investigate methods to vendor python3 compatible packages

Status: RESOLVED INVALID

(Firefox Build System :: General, enhancement)

Description

[Edwin Takahashi (:egao | infrequent contributor)]

./mach vendor should also vendor the python 3 packages of the ones currently in the list.

Comment 1

[Edwin Takahashi (:egao | infrequent contributor)]

Not sure what the best approach here would be. I recall running into issues with the vendored voluptuous module while working on bug 1577883...but do not recall why. It could be that I was imagining voluptuous being at fault in a post-partum sleep deprivation haze.

Anyway, assuming some actions need to be taken, it is a fact that ./mach vendor python script relies heavily on mozbuild which is a behemoth.

I imagine two paths, but am open to other ideas and/or corrections as I do not know enough about the vendoring process:

1. create a parallel ./mach vendor python3 command that would install, well, python3 equivalent of the current vendored modules.
2. update each module to the last version that maintains python2.7/python3.5 compatibility. For example, with pytest it would be 4.6.5.

What might be the best way forward? I think we will be able to migrate some more mach commands to python3 if vendored modules support both pythons.

Comment 2

[Andrew Halberstadt [:ahal]]

I think my preferred approach would be to tackle this one module at a time as needed. For example, if we decide to convert ./mach doc and it depends on a vendored package that only supports Python 2, then figure it out at that time. Either by modifying ./mach doc to not use that dependency, by vendoring a newer version (which theoretically supports Python 3), or vendoring a second copy (if there are independent implementations of the library).

I think trying to make all vendored modules compatible with both Py2/Py3 upfront will end up wasting a bunch of time, as we might not need to convert them all. It will also be very hard as we will likely break the world if we do this.

It's also worth noting that ./mach vendor python isn't actually necessary to vendor things. It's more of a convenience script. So I might not even bother trying to make it support both (i.e, let's just make sure it works with Python 3 and drop Python 2 support). Then if someone absolutely needs to vendor a Python 2-only library, they can still do so manually.

Comment 3

[Edwin Takahashi (:egao | infrequent contributor)]

:ahal - thanks, that sounds like a reasonable way forward. After writing the initial comment, I checked the version of some of the modules that we vendor, and it does appear that most of them are relatively modern. My initial assumption had been that we vendor a much older version for many of the libraries but that has been shown to be incorrect.

As for ./mach vendor python script, I looked into supporting python3 but since it inherits from Mozbuild, it doesn't appear possible to update just the vendor scripts alone unless it is rewritten to decouple it from the build system. Since it's a convenience script it should not be so tightly coupled to Mozbuild, is my opinion.

This realization is what prompted my comment in slack that we should be looking to take out as many things as possible.

Comment 4

[Edwin Takahashi (:egao | infrequent contributor)]

Now that I've worked a bit more with vendored packages, it does look like we've already vendored python3 compatible packages for all existing packages; additionally, that this is simply a helper script to manage the vendoring process.

I think we should look into migrating to running the helper script with python3 as well as using a more modern pip version, but that's a story for another day and another bug.

This bug can be closed as invalid since the assertions made in the first comment is in fact incorrect.