1659593 - Make `mach vendor python` produce cross-platform, cross-Python-version results

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Ricky Stewart]

The vendored glean_parser package has a conditional dependency which may or may not actually be a dependency depending on the version of Python. In particular glean_parser only depends on iso8601 for Python 3.6.

Because mach vendor python uses pip-compile to generate requirements.txt (which will use the user's Python 3), that makes the output of mach vendor python nondeterministic depending on whether you're running Python 3.6 or Python 3.7+.

This is a known issue with pip-compile, and there's apparently no plan to support "cross-platform" requirements.txt generation, either in the form of generating a single requirements.txt file that supports all versions of Python between 3.6 and 3.8, or by "cross-compiling" to a different version of Python than your version of pip-compile.

Dependencies can be conditional on a large number of environmental properties, including the OS. This means that conceptually, the only "correct" thing to do would be to generate one requirements.txt file for every environment -- here, that means every (Python 3 version * OS combination). Since pip-compile can't "cross-compile", we'd have to run pip-compile once in every environment, which is not tractable. It's still only very dubiously doable if we were to restrict to one Python 3 version, like mandating the use of e.g. Python 3.7.X or whatever.

For now, the use of conditional dependencies via environment markers doesn't seem to be prohibitive and we can probably continue to get by with careful use of mach vendor python and careful pruning of the changes it introduces, but I'm not sure whether that's a sustainable long-term solution.

Comment 1

[Andrew Halberstadt [:ahal]]

Yeah, this has been an issue since the dawn of vendoring in m-c. We've always sort of worked around it ad-hoc (when a dev complains, add the missing thing).

Pip-tools maintainer has explicitly stated that it's designed to work with the current environment and generating a single requirements.txt that can support multiple environments is out of scope.

So I'm not sure we can fix this properly, but here are few easy mitigation steps:

1. Install things to the new global venvs rather than vendoring
2. Ensure |mach vendor python| can only run with our minimum supported version of Python (3.6), since I suspect backport deps are the common failure case here.

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Today's current state is that we enforce in CI that the vendored code matches the results of ./mach vendor python on Python 3.6 with Linux.

Note that the state-of-the-art here has drastically improved since this was discussed last year. We can use the same strategy as we are for lockfiles: Use poetry to generate lockfiles with correctly-managed environment markers.

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1659593: Make `./mach vendor python` cross-platform

Use poetry instead of pip-compile, then leverage pip to assert
that there's no conflicts between environments.

This enables simplifying requirements.in, since colorama is now
automatically picked up, as vendoring is no longer linux-py3.6-specific.

poetry==1.2.0a2 was chosen because it was the first version that would
include pip/setuptools/wheel in the requirements.txt, yet it was
also the last version that supports Python 3.6 (as still used in CI in
some spots).

Sorry about the requirements.txt noise, that's because poetry doesn't
sort hashes yet [1]. Fortunately, you can be confident in
requirements.txt changes by checking that this patch doesn't include
changes to vendored packages themselves (I'm including the results of a
./mach vendor python).

[1] https://github.com/python-poetry/poetry/issues/5408

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/185c467d1e4b
Make `./mach vendor python` cross-platform r=ahal

Comment 5

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/185c467d1e4b