Closed Bug 1602771 Opened 3 years ago Closed 2 years ago

`mach vendor python` is broken (AttributeError: module 'pip._internal.download' has no attribute 'is_file_url')

Categories

(Firefox Build System :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1602271

People

(Reporter: Dexter, Assigned: rstewart)

References

(Blocks 1 open bug)

Details

(Keywords: in-triage)

Attachments

(1 file, 3 obsolete files)

Attached file error_output.txt

I'm trying to vendor glean-parser==1.14.0 on Windows, a Python 3.5 library from Mozilla that we need for telemetry with the following command:

`./mach vendor python glean-parser==1.14.0

And I'm receiving this error:

AttributeError: module 'pip._internal.download' has no attribute 'is_file_url'
Error running mach:

['vendor', 'python', 'glean-parser==1.14.0']

The full output is in the attachment.

A colleague tried that on Mac and it resulted in:

ModuleNotFoundError: No module named 'piptools'
Error running mach:
['vendor', 'python', 'glean-parser==1.14.0']

Is this a known issue? I could not find any related bug already filed.

Blocks: 1602773

From this issue this seems to be a version compatibility problem.

Assignee: nobody → rstewart

The GitHub issue says pip 19.3 is compatible with pip-tools 4.2.0. We are running two versions of pip in-tree: 19.1 and 19.3. We are using pip-tools 3.9.0.

Pip is being installed from the virtualenv package, I don't know how it chooses the pip version it installs on the host. But I think it is possible to get a combination of pip-tools 3.9.0 and pip 19.3, which is a problematic combination according to the GitHub issue.

(In reply to Māris Fogels [:mars] (please needinfo) from comment #2)

Pip is being installed from the virtualenv package, I don't know how it chooses the pip version it installs on the host. But I think it is possible to get a combination of pip-tools 3.9.0 and pip 19.3, which is a problematic combination according to the GitHub issue.

Let me know if there's something I can do to help diagnosing this, I can reproduce it.

There is a copy of the Python 3 pip tool available in the init_py3 virtualenv in the object directory. Can you post the output of the pip list command?

Here is the command path on my Linux system:

 ~/w/firefox  obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/bin/pip list
Flags: needinfo?(alessio.placitelli)

(In reply to Māris Fogels [:mars] (please needinfo) from comment #4)

There is a copy of the Python 3 pip tool available in the init_py3 virtualenv in the object directory. Can you post the output of the pip list command?

Here is the command path on my Linux system:

 ~/w/firefox  obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/bin/pip list

I'm on Windows 10, for me pip was in the /Scripts directory, and I had no /bin

$ obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip list
Package        Version
-------------- ---------
certifi        2018.4.16
enum34         1.1.6
hpack          3.0.0
hyperframe     5.1.0
mock           1.0.0
pip            19.3.1
pip-tools      3.9.0
psutil         5.4.3
pyasn1         0.3.7
pyasn1-modules 0.1.5
redo           2.0.3
requests       2.9.1
rsa            3.1.4
setuptools     41.6.0
webencodings   0.5.1
wheel          0.33.6
Flags: needinfo?(alessio.placitelli)

Is there anything else I can do to help debug this?

Flags: needinfo?(mars)

(In reply to Alessio Placitelli [:Dexter] from comment #6)

Is there anything else I can do to help debug this?

If you have a clear set of steps to reproduce this bug on your system you could help verify a possible fix.

After triggering this issue could you try upgrading pip-tools in the virtualenv and re-run the steps that should fail? If it's a version mismatch then bumping pip-tools to v4+ should fix things.

$ obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip list | grep pip-tools
# verify pip-tools v3.9.0
# reproduce error

$  obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip install -U pip-tools
$ obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip list
# verify pip-tools v4+
# attempt to reproduce error again
Flags: needinfo?(mars) → needinfo?(alessio.placitelli)

(In reply to Māris Fogels [:mars] (please needinfo) from comment #7)

(In reply to Alessio Placitelli [:Dexter] from comment #6)
After triggering this issue could you try upgrading pip-tools in the virtualenv and re-run the steps that should fail? If it's a version mismatch then bumping pip-tools to v4+ should fix things.

$  obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip install -U pip-tools
$ obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/pip list
# verify pip-tools v4+
# attempt to reproduce error again

Unfortunately, following the above steps fails because mach attempts to re-install pip-tools 3.9.0 from third_part.

$ ./mach vendor python glean-parser==1.15.5
Processing .\third_party\python\pip-tools
Building wheels for collected packages: pip-tools
Building wheel for pip-tools (setup.py) ... done
Created wheel for pip-tools: filename=pip_tools-3.9.0-py2.py3-none-any.whl size=39969 sha256=25fbba046c7c926057215fb9cb05aa78c706ccc88272384289e9ba412a79a75a
Stored in directory: C:\Users\Alessio\AppData\Local\pip\Cache\wheels\34\e5\a8\db5e6cebbefc9cf64ad80b2abd05ca0ccc5416dbc6a52643e0
Successfully built pip-tools
Installing collected packages: pip-tools
Found existing installation: pip-tools 4.3.0
Uninstalling pip-tools-4.3.0:
Successfully uninstalled pip-tools-4.3.0
Successfully installed pip-tools-3.9.0

I tried to be brave and manually vendor pip-tools-4.3.0 (copy the source code in third_party/python/pip-tools) and then call mach vendor python, but it failed with this new error, after downloading a bunch of libraries:

ERROR: Command errored out with exit status 1: 'e:\mozilla-unified\obj-x86_64-pc-mingw32_virtualenvs\init_py3\scripts\python3.exe' 'e:\mozilla-unified\obj-x86_64-pc-mingw32_virtualenvs\init_py3\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\Alessio\AppData\Local\Temp\pip-build-env-4lvbi3if\overlay' --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'pip>=18' 'setuptools>=40' wheel Check the logs for full command output.

I'm not really sure where to look up the full command output, though.

Flags: needinfo?(alessio.placitelli)
Flags: needinfo?(mars)
Flags: needinfo?(mars) → needinfo?(rstewart)

FWIW, I hit the same bug on macos 14 with homebrew. Maybe this is similar to bug 1607470, where we pick up the wrong python binary in some way?

(In reply to Axel Hecht [:Pike] from comment #9)

FWIW, I hit the same bug on macos 14 with homebrew. Maybe this is similar to bug 1607470, where we pick up the wrong python binary in some way?

Not sure: the other bug was merged, I just tested locally and I still have the same problem :(

OK, so I haven't found an in-tree version of pip, but I did update pip-tools to 4.40 and that worked.

Once this ran, it did add quite a few packages to thirdparty that I didn't expect, like aiohttp, async-timeout, chardet, idna, multidict, yarl.

But that might be just me successfully running ./mach vendor python on py3, looking at https://searchfox.org/mozilla-central/source/third_party/python/taskcluster/setup.py#68

Yep, it's vendoring dependencies that exist on Python 3 but not 2.

I think we should:

  1. Remove all packages only needed by Python 2 from the requirements.in
  2. Modify mach vendor python to bail out if it gets called from Python 2
  3. Handle Python 2 vendoring by hand from now on (which we're already doing for both atm anyway)

That way we can at least use it with Python 3 and eventually the need to support Python 2 vendoring will disappear.

Is there anything we could do, in the short run, to vendor the Python 3 library we need?

Flags: needinfo?(ahal)

Thanks to the herculean effort by Jannis (:jezdez), we were able to identify the culprit of this. It's a chicken and egg problem :-) "pip-tools" needs to be updated and vendored, but vendoring requires pip-tools.

As outlined in comment 8, I had already tried fixing this manually, but I had failed. Jannis made me perform the following process, which worked:

  • replace the directory of the vendored version of pip-tools (3.9.0) with the source from pip-tools 4.4.1
  • manually change the version of pip-tools to 4.4.1 in third_party/python/requirements.in
  • run ./mach vendor python

This makes the command I needed to run work. I will provide a patch with these changes.

Flags: needinfo?(rstewart)
Flags: needinfo?(ahal)
Attachment #9124565 - Attachment is obsolete: true

Depends on D62042

Depends on: 1601140
Attachment #9125073 - Attachment is obsolete: true
Attachment #9125066 - Attachment is obsolete: true

This is now working for me, after bug 1602271 updated pip-tools.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1602271
You need to log in before you can comment on or make changes to this bug.