1601140 - |mach vendor python| wants to remove Python 2 packages from requirements.txt

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, defect, P3)

Description

[Māris Fogels [:mars] (please needinfo)]

mach vendor python is trying to remove Python 2 dependencies from third_party/python/requirements.txt.

STR:

1. Run mach vendor python for an existing library - it should be a no-op. e.g. ./mach vendor python pip-tools==3.0.0.
2. Inspect the requirements.txt changes: hg diff third_party/python/requirements.txt.

Expected:
There should be no changes.

Actual:
The Python 2 libraries, such as enum34, have been removed.

Here's a fragment of the diff that was generated:

@@ -25,16 +57,13 @@ cookies==2.2.1 \
 distro==1.4.0 \
     --hash=sha256:362dde65d846d23baee4b5c058c8586f219b5a54be1cf5fc6ff55c4578392f57 \
     --hash=sha256:eedf82a470ebe7d010f1872c17237c79ab04097948800029994fa458e52fb4b4
-enum34==1.1.6 \
-    --hash=sha256:2d81cbbe0e73112bdfe6ef8576f2238f2ba27dd0d55752a776c41d38b7da2850 \
-    --hash=sha256:644837f692e5f550741432dd3f223bbb9852018674981b1664e5dc339387588a \
-    --hash=sha256:6bd0f6ad48ec2aa117d3d141940d484deccda84d4fcd884f5c3d93c23ecd8c79 \
-    --hash=sha256:8ad8c4783bf61ded74527bffb48ed9b54166685e4230386a9ed9b1279e2df5b1 \
-    # via mozilla-version
-funcsigs==1.0.2 \
-    --hash=sha256:330cc27ccbf7f1e992e69fef78261dc7c6569012cf397db8d3de0234e6c937ca \
-    --hash=sha256:a7bb0f2cf3a3fd1ab2732cb49eba4252c2af4240442415b4abce3b87022a8f50 \
-    # via pytest

If we look at the mozilla-version package requirements we can see that enum34 is only required for Python < 3.4.

third_party/python/mozilla-version/requirements.txt.in

# pyup: ignore file
enum34; python_version < '3.4'
attrs

mach vendor runs with the Python 3.5+ since https://hg.mozilla.org/mozilla-central/rev/27c058fff29bcefbf40f81b82c8f990bf01ef5be. It could be that pip-tools is checking sys.version_info, finding it's running under Python 3.4+, and removing the Python 2-only requirement.

I tried to verify the problem changeset but bisection of older commits of mach vendor on my system keeps failing with program errors.

Comment 1

[Māris Fogels [:mars] (please needinfo)]

ahal and I discussed solutions for this bug. We can change the in-tree package management so that mach vendor python manages only Python 3 packages and remove the automated Python 2 package management support entirely. The Python 2 support libraries can remain in-tree and available to Python 2 code via our virtualenv site customization.

There is a risk that Python 3 libraries get upgraded with mach vendor python and drop API compatibility with Python 2 libraries, breaking the Python 2 part of the tree, but that's a risk only as long as the 2-to-3 migration is taking place.

To drop Python 2 support from mach vendor python we need to remove the Python 2 packages from requirements.in and add the sources for the packages to the third_party/python package directory. If the Python 2 packages are missing from third_party/python/requirements.in then mach vendor python should not see or touch them.

Comment 2

[Axel Hecht [:Pike]]

Taking a look at requirements.txt, it lists dependencies of things in requirements.in.

Which makes me think that we'll need to disable/comment out any packages in requirements.in that have python-version-dependent dependencies? And then mach vendor shouldn't touch those?

Comment 3

[Andrew Halberstadt [:ahal]]

Correct.

Maybe we could have a py2requirements.in/txt and py3requirements.in/txt. Then when we run ./mach vendor python, it uses what ever file matches the current executable (or supports a --python flag).

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1601140 - Support line comments in third_party/python/requirements.in, r?rstewart

Depends on D64755

Comment 5

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1601140 - Remove 'taskcluster' and 'taskcluster-urls' from third_party/python/requirements.in, r?rstewart

This adds a warning not to add any Python 2 packages to the list. From now on,
it will be for things used by Python 3 only and Python 2 needs to be vendored
by hand.

Depends on D64756

Comment 6

[Andrew Halberstadt [:ahal]]

So while I think the patches here are necessary, it turns out mach vendor python still tries to add aiohttp et al, even though they are no longer in the requirements.txt... More investigation needed.

Comment 7

[Andrew Halberstadt [:ahal]]

Oh nvm, I just add those packages as untracked files in my repo (probably from some previous invocation of mach vendor python before these patches. There's a call to repository.add_remove_files that was adding them back to my commit each time. After a purge the issue was resolved.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/27532e800952
Support line comments in third_party/python/requirements.in, r=rstewart
https://hg.mozilla.org/integration/autoland/rev/2d5b526e781e
Remove 'taskcluster' and 'taskcluster-urls' from third_party/python/requirements.in, r=rstewart

Comment 9

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/27532e800952
https://hg.mozilla.org/mozilla-central/rev/2d5b526e781e