1602771 - `mach vendor python` is broken (AttributeError: module 'pip._internal.download' has no attribute 'is_file_url')

Status: RESOLVED DUPLICATE of bug 1602271

(Firefox Build System :: General, defect)

Description

[Alessio Placitelli [:Dexter]]

Attachment: error_output.txt

I'm trying to vendor glean-parser==1.14.0 on Windows, a Python 3.5 library from Mozilla that we need for telemetry with the following command:

`./mach vendor python glean-parser==1.14.0

And I'm receiving this error:

AttributeError: module 'pip._internal.download' has no attribute 'is_file_url'
Error running mach:

['vendor', 'python', 'glean-parser==1.14.0']

The full output is in the attachment.

A colleague tried that on Mac and it resulted in:

ModuleNotFoundError: No module named 'piptools'
Error running mach:
['vendor', 'python', 'glean-parser==1.14.0']

Is this a known issue? I could not find any related bug already filed.

Comment 1

[Alessio Placitelli [:Dexter]]

From this issue this seems to be a version compatibility problem.

Comment 2

[Māris Fogels [:mars] (please needinfo)]

The GitHub issue says pip 19.3 is compatible with pip-tools 4.2.0. We are running two versions of pip in-tree: 19.1 and 19.3. We are using pip-tools 3.9.0.

Pip is being installed from the virtualenv package, I don't know how it chooses the pip version it installs on the host. But I think it is possible to get a combination of pip-tools 3.9.0 and pip 19.3, which is a problematic combination according to the GitHub issue.

Comment 3

[Alessio Placitelli [:Dexter]]

(In reply to Māris Fogels [:mars] (please needinfo) from comment #2)

Pip is being installed from the virtualenv package, I don't know how it chooses the pip version it installs on the host. But I think it is possible to get a combination of pip-tools 3.9.0 and pip 19.3, which is a problematic combination according to the GitHub issue.

Let me know if there's something I can do to help diagnosing this, I can reproduce it.

Comment 4

[Māris Fogels [:mars] (please needinfo)]

There is a copy of the Python 3 pip tool available in the init_py3 virtualenv in the object directory. Can you post the output of the pip list command?

Here is the command path on my Linux system:

 ~/w/firefox  obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/bin/pip list

Comment 5

[Alessio Placitelli [:Dexter]]

(In reply to Māris Fogels [:mars] (please needinfo) from comment #4)

There is a copy of the Python 3 pip tool available in the init_py3 virtualenv in the object directory. Can you post the output of the pip list command?

Here is the command path on my Linux system:

 ~/w/firefox  obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/bin/pip list

I'm on Windows 10, for me pip was in the /Scripts directory, and I had no /bin

$ obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip list
Package        Version
-------------- ---------
certifi        2018.4.16
enum34         1.1.6
hpack          3.0.0
hyperframe     5.1.0
mock           1.0.0
pip            19.3.1
pip-tools      3.9.0
psutil         5.4.3
pyasn1         0.3.7
pyasn1-modules 0.1.5
redo           2.0.3
requests       2.9.1
rsa            3.1.4
setuptools     41.6.0
webencodings   0.5.1
wheel          0.33.6

Comment 6

[Alessio Placitelli [:Dexter]]

Is there anything else I can do to help debug this?

Comment 7

[Māris Fogels [:mars] (please needinfo)]

(In reply to Alessio Placitelli [:Dexter] from comment #6)

Is there anything else I can do to help debug this?

If you have a clear set of steps to reproduce this bug on your system you could help verify a possible fix.

After triggering this issue could you try upgrading pip-tools in the virtualenv and re-run the steps that should fail? If it's a version mismatch then bumping pip-tools to v4+ should fix things.

$ obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip list | grep pip-tools
# verify pip-tools v3.9.0
# reproduce error

$  obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip install -U pip-tools
$ obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip list
# verify pip-tools v4+
# attempt to reproduce error again

Comment 8

[Alessio Placitelli [:Dexter]]

(In reply to Māris Fogels [:mars] (please needinfo) from comment #7)

(In reply to Alessio Placitelli [:Dexter] from comment #6)
After triggering this issue could you try upgrading pip-tools in the virtualenv and re-run the steps that should fail? If it's a version mismatch then bumping pip-tools to v4+ should fix things.

$  obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip install -U pip-tools
$ obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip list
# verify pip-tools v4+
# attempt to reproduce error again

Unfortunately, following the above steps fails because mach attempts to re-install pip-tools 3.9.0 from third_part.

$ ./mach vendor python glean-parser==1.15.5
Processing .\third_party\python\pip-tools
Building wheels for collected packages: pip-tools
Building wheel for pip-tools (setup.py) ... done
Created wheel for pip-tools: filename=pip_tools-3.9.0-py2.py3-none-any.whl size=39969 sha256=25fbba046c7c926057215fb9cb05aa78c706ccc88272384289e9ba412a79a75a
Stored in directory: C:\Users\Alessio\AppData\Local\pip\Cache\wheels\34\e5\a8\db5e6cebbefc9cf64ad80b2abd05ca0ccc5416dbc6a52643e0
Successfully built pip-tools
Installing collected packages: pip-tools
Found existing installation: pip-tools 4.3.0
Uninstalling pip-tools-4.3.0:
Successfully uninstalled pip-tools-4.3.0
Successfully installed pip-tools-3.9.0

I tried to be brave and manually vendor pip-tools-4.3.0 (copy the source code in third_party/python/pip-tools) and then call mach vendor python, but it failed with this new error, after downloading a bunch of libraries:

ERROR: Command errored out with exit status 1: 'e:\mozilla-unified\obj-x86_64-pc-mingw32_virtualenvs\init_py3\scripts\python3.exe' 'e:\mozilla-unified\obj-x86_64-pc-mingw32_virtualenvs\init_py3\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\Alessio\AppData\Local\Temp\pip-build-env-4lvbi3if\overlay' --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'pip>=18' 'setuptools>=40' wheel Check the logs for full command output.

I'm not really sure where to look up the full command output, though.

Comment 9

[Axel Hecht [:Pike]]

FWIW, I hit the same bug on macos 14 with homebrew. Maybe this is similar to bug 1607470, where we pick up the wrong python binary in some way?

Comment 10

[Alessio Placitelli [:Dexter]]

(In reply to Axel Hecht [:Pike] from comment #9)

FWIW, I hit the same bug on macos 14 with homebrew. Maybe this is similar to bug 1607470, where we pick up the wrong python binary in some way?

Not sure: the other bug was merged, I just tested locally and I still have the same problem :(

Comment 11

[Axel Hecht [:Pike]]

OK, so I haven't found an in-tree version of pip, but I did update pip-tools to 4.40 and that worked.

Once this ran, it did add quite a few packages to thirdparty that I didn't expect, like aiohttp, async-timeout, chardet, idna, multidict, yarl.

But that might be just me successfully running ./mach vendor python on py3, looking at https://searchfox.org/mozilla-central/source/third_party/python/taskcluster/setup.py#68

Comment 12

[Andrew Halberstadt [:ahal]]

Yep, it's vendoring dependencies that exist on Python 3 but not 2.

I think we should:

1. Remove all packages only needed by Python 2 from the requirements.in
2. Modify mach vendor python to bail out if it gets called from Python 2
3. Handle Python 2 vendoring by hand from now on (which we're already doing for both atm anyway)

That way we can at least use it with Python 3 and eventually the need to support Python 2 vendoring will disappear.

Comment 13

[Alessio Placitelli [:Dexter]]

Is there anything we could do, in the short run, to vendor the Python 3 library we need?

Comment 14

[Alessio Placitelli [:Dexter]]

Thanks to the herculean effort by Jannis (:jezdez), we were able to identify the culprit of this. It's a chicken and egg problem :-) "pip-tools" needs to be updated and vendored, but vendoring requires pip-tools.

As outlined in comment 8, I had already tried fixing this manually, but I had failed. Jannis made me perform the following process, which worked:

• replace the directory of the vendored version of pip-tools (3.9.0) with the source from pip-tools 4.4.1
• manually change the version of pip-tools to 4.4.1 in third_party/python/requirements.in
• run ./mach vendor python

This makes the command I needed to run work. I will provide a patch with these changes.

Comment 15

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1602771 - Update pip-tools to version 4.4.1

Comment 16

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1602771 - Update pip-tools to version 4.4.1

Comment 17

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1602771 - Update the python vendored dependencies

Depends on D62042

Comment 18

[Alessio Placitelli [:Dexter]]

This is now working for me, after bug 1602271 updated pip-tools.