`mach vendor python` is broken (AttributeError: module 'pip._internal.download' has no attribute 'is_file_url')
Categories
(Firefox Build System :: General, defect)
Tracking
(Not tracked)
People
(Reporter: Dexter, Assigned: rstewart)
References
(Blocks 1 open bug)
Details
(Keywords: in-triage)
Attachments
(1 file, 3 obsolete files)
4.10 KB,
text/plain
|
Details |
I'm trying to vendor glean-parser==1.14.0
on Windows, a Python 3.5 library from Mozilla that we need for telemetry with the following command:
`./mach vendor python glean-parser==1.14.0
And I'm receiving this error:
AttributeError: module 'pip._internal.download' has no attribute 'is_file_url'
Error running mach:['vendor', 'python', 'glean-parser==1.14.0']
The full output is in the attachment.
A colleague tried that on Mac and it resulted in:
ModuleNotFoundError: No module named 'piptools'
Error running mach:
['vendor', 'python', 'glean-parser==1.14.0']
Is this a known issue? I could not find any related bug already filed.
Reporter | ||
Comment 1•3 years ago
|
||
From this issue this seems to be a version compatibility problem.
Assignee | ||
Updated•3 years ago
|
Comment 2•3 years ago
|
||
The GitHub issue says pip 19.3 is compatible with pip-tools 4.2.0. We are running two versions of pip in-tree: 19.1 and 19.3. We are using pip-tools 3.9.0.
Pip is being installed from the virtualenv package, I don't know how it chooses the pip version it installs on the host. But I think it is possible to get a combination of pip-tools 3.9.0 and pip 19.3, which is a problematic combination according to the GitHub issue.
Reporter | ||
Comment 3•3 years ago
|
||
(In reply to Māris Fogels [:mars] (please needinfo) from comment #2)
Pip is being installed from the virtualenv package, I don't know how it chooses the pip version it installs on the host. But I think it is possible to get a combination of pip-tools 3.9.0 and pip 19.3, which is a problematic combination according to the GitHub issue.
Let me know if there's something I can do to help diagnosing this, I can reproduce it.
Comment 4•3 years ago
|
||
There is a copy of the Python 3 pip tool available in the init_py3
virtualenv in the object directory. Can you post the output of the pip list
command?
Here is the command path on my Linux system:
~/w/firefox obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/bin/pip list
Updated•3 years ago
|
Reporter | ||
Comment 5•3 years ago
|
||
(In reply to Māris Fogels [:mars] (please needinfo) from comment #4)
There is a copy of the Python 3 pip tool available in the
init_py3
virtualenv in the object directory. Can you post the output of thepip list
command?Here is the command path on my Linux system:
~/w/firefox obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/bin/pip list
I'm on Windows 10, for me pip
was in the /Scripts
directory, and I had no /bin
$ obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip list
Package Version
-------------- ---------
certifi 2018.4.16
enum34 1.1.6
hpack 3.0.0
hyperframe 5.1.0
mock 1.0.0
pip 19.3.1
pip-tools 3.9.0
psutil 5.4.3
pyasn1 0.3.7
pyasn1-modules 0.1.5
redo 2.0.3
requests 2.9.1
rsa 3.1.4
setuptools 41.6.0
webencodings 0.5.1
wheel 0.33.6
Reporter | ||
Comment 6•3 years ago
|
||
Is there anything else I can do to help debug this?
Comment 7•3 years ago
|
||
(In reply to Alessio Placitelli [:Dexter] from comment #6)
Is there anything else I can do to help debug this?
If you have a clear set of steps to reproduce this bug on your system you could help verify a possible fix.
After triggering this issue could you try upgrading pip-tools in the virtualenv and re-run the steps that should fail? If it's a version mismatch then bumping pip-tools to v4+ should fix things.
$ obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip list | grep pip-tools
# verify pip-tools v3.9.0
# reproduce error
$ obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip install -U pip-tools
$ obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip list
# verify pip-tools v4+
# attempt to reproduce error again
Reporter | ||
Comment 8•3 years ago
|
||
(In reply to Māris Fogels [:mars] (please needinfo) from comment #7)
(In reply to Alessio Placitelli [:Dexter] from comment #6)
After triggering this issue could you try upgrading pip-tools in the virtualenv and re-run the steps that should fail? If it's a version mismatch then bumping pip-tools to v4+ should fix things.$ obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip install -U pip-tools $ obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip list # verify pip-tools v4+ # attempt to reproduce error again
Unfortunately, following the above steps fails because mach
attempts to re-install pip-tools 3.9.0 from third_part.
$ ./mach vendor python glean-parser==1.15.5
Processing .\third_party\python\pip-tools
Building wheels for collected packages: pip-tools
Building wheel for pip-tools (setup.py) ... done
Created wheel for pip-tools: filename=pip_tools-3.9.0-py2.py3-none-any.whl size=39969 sha256=25fbba046c7c926057215fb9cb05aa78c706ccc88272384289e9ba412a79a75a
Stored in directory: C:\Users\Alessio\AppData\Local\pip\Cache\wheels\34\e5\a8\db5e6cebbefc9cf64ad80b2abd05ca0ccc5416dbc6a52643e0
Successfully built pip-tools
Installing collected packages: pip-tools
Found existing installation: pip-tools 4.3.0
Uninstalling pip-tools-4.3.0:
Successfully uninstalled pip-tools-4.3.0
Successfully installed pip-tools-3.9.0
I tried to be brave and manually vendor pip-tools-4.3.0
(copy the source code in third_party/python/pip-tools
) and then call mach vendor python
, but it failed with this new error, after downloading a bunch of libraries:
ERROR: Command errored out with exit status 1: 'e:\mozilla-unified\obj-x86_64-pc-mingw32_virtualenvs\init_py3\scripts\python3.exe' 'e:\mozilla-unified\obj-x86_64-pc-mingw32_virtualenvs\init_py3\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\Alessio\AppData\Local\Temp\pip-build-env-4lvbi3if\overlay' --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'pip>=18' 'setuptools>=40' wheel Check the logs for full command output.
I'm not really sure where to look up the full command output, though.
Reporter | ||
Updated•3 years ago
|
Reporter | ||
Updated•3 years ago
|
Comment 9•3 years ago
|
||
FWIW, I hit the same bug on macos 14 with homebrew. Maybe this is similar to bug 1607470, where we pick up the wrong python binary in some way?
Reporter | ||
Comment 10•3 years ago
|
||
(In reply to Axel Hecht [:Pike] from comment #9)
FWIW, I hit the same bug on macos 14 with homebrew. Maybe this is similar to bug 1607470, where we pick up the wrong python binary in some way?
Not sure: the other bug was merged, I just tested locally and I still have the same problem :(
Comment 11•3 years ago
|
||
OK, so I haven't found an in-tree version of pip, but I did update pip-tools
to 4.40 and that worked.
Once this ran, it did add quite a few packages to thirdparty that I didn't expect, like aiohttp, async-timeout, chardet, idna, multidict, yarl.
But that might be just me successfully running ./mach vendor python
on py3, looking at https://searchfox.org/mozilla-central/source/third_party/python/taskcluster/setup.py#68
Comment 12•3 years ago
|
||
Yep, it's vendoring dependencies that exist on Python 3 but not 2.
I think we should:
- Remove all packages only needed by Python 2 from the
requirements.in
- Modify
mach vendor python
to bail out if it gets called from Python 2 - Handle Python 2 vendoring by hand from now on (which we're already doing for both atm anyway)
That way we can at least use it with Python 3 and eventually the need to support Python 2 vendoring will disappear.
Reporter | ||
Comment 13•3 years ago
|
||
Is there anything we could do, in the short run, to vendor the Python 3 library we need?
Reporter | ||
Comment 14•3 years ago
|
||
Thanks to the herculean effort by Jannis (:jezdez), we were able to identify the culprit of this. It's a chicken and egg problem :-) "pip-tools" needs to be updated and vendored, but vendoring requires pip-tools.
As outlined in comment 8, I had already tried fixing this manually, but I had failed. Jannis made me perform the following process, which worked:
- replace the directory of the vendored version of pip-tools (3.9.0) with the source from pip-tools 4.4.1
- manually change the version of pip-tools to 4.4.1 in third_party/python/requirements.in
- run ./mach vendor python
This makes the command I needed to run work. I will provide a patch with these changes.
Reporter | ||
Comment 15•3 years ago
|
||
Updated•3 years ago
|
Reporter | ||
Comment 16•3 years ago
|
||
Reporter | ||
Comment 17•3 years ago
|
||
Depends on D62042
Updated•3 years ago
|
Updated•3 years ago
|
Reporter | ||
Comment 18•2 years ago
|
||
This is now working for me, after bug 1602271 updated pip-tools.
Description
•