1782785 - Error running `./mach vendor python`

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, defect, P2)

Description

[Andrew Halberstadt [:ahal]]

When I run ./mach vendor python (even on a clean tree) I get:

$ ./mach vendor python
created virtual environment CPython3.10.5.final.0-64 in 17ms
  creator CPython3Posix(dest=/home/ahal/dev/mozilla-unified/obj-x86_64-pc-linux-gnu/_virtualenvs/vendor, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
Collecting poetry==1.2.0a2
  Using cached poetry-1.2.0a2-py3-none-any.whl (190 kB)
 
<snip>

Collecting jsmin==2.1.0
  Using cached jsmin-2.1.0.tar.gz (9.6 kB)
Collecting json-e==2.7.0
  Using cached json-e-2.7.0.tar.gz (17 kB)
Collecting jsonschema==3.2.0
  Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
ERROR: Could not find a version that satisfies the requirement looseversion==1.0.1markupsafe==1.1.1 (from versions: 1.0.0, 1.0.1)
ERROR: No matching distribution found for looseversion==1.0.1markupsafe==1.1.1
Error running mach:

    ['vendor', 'python']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file vendor| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/home/ahal/dev/mozilla-unified/obj-x86_64-pc-linux-gnu/_virtualenvs/vendor/bin/python', '-m', 'pip', 'download', '-r', '/tmp/tmpc8w78ai_/requirements.no-markers.txt', '--no-deps', '--dest', '/tmp/tmpi1j4uks3', '--abi', 'none', '--platform', 'any']' returned non-zero exit status 1.

  File "/home/ahal/dev/mozilla-unified/python/mozbuild/mozbuild/vendor/mach_commands.py", line 229, in vendor_python
    vendor_command.vendor(keep_extra_files)
  File "/home/ahal/dev/mozilla-unified/python/mozbuild/mozbuild/vendor/vendor_python.py", line 54, in vendor
    subprocess.check_call(
  File "/home/ahal/.pyenv/versions/3.10.5/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)

Note for some reason the looseversion and markupsafe packages are concatenated together. Sylvestre also hit this error but with two different packages than me.

Comment 1

[Andrew Halberstadt [:ahal]]

This specific error is happening because the looseversion package doesn't have any hashes and the parsing logic that removes environment markers is tripping up over this fact. I have a patch to make the parsing logic more robust, but that just changes the error to:

ERROR: Hashes are required in --require-hashes mode, but they are missing from some requirements. Here is a list of those requirements along with the hashes their downloaded archives actually had. Add lines like these to your requirements files to prevent tampering. (If you did not enable --require-hashes manually, note that it turns on automatically when any package has a hash.)
    looseversion==1.0.1 --hash=sha256:a205beabd0ffd40488edb9ccb3a39134510fc7c0c2847a25079f559e59c004ac

For whatever reason Poetry does not pin the hashes of the looseversion package. I notice that this dependency was added a few days ago in bug 1777414. This looks like an upstream Poetry problem, so I filed an issue.

Alex, fyi mach vendor is now broken. I'm still not 100% sure what's going on, but we may want to consider backing out your patch (I can confirm that upgrading to the parent of your revision "fixes" mach vendor python). Though this definitely isn't your fault or anything.. I'm just not sure how to fix this :).

Comment 2

[Andrew Halberstadt [:ahal]]

Thanks to a tip from someone in the poetry issue, clearing the cache seemed to work:

$ poetry cache clear pypi --all

Though now I'm hitting the same error that Sylvestre was hitting ><:

Writing lock file
ERROR: Could not find a version that satisfies the requirement aiohttp==3.7.4.post0appdirs==1.4.4 (from versions ...)
ERROR: No matching distribution found for aiohttp==3.7.4.post0appdirs==1.4.4
Error running mach:

    ['vendor', 'python']

I guess it's still progress?

Comment 3

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

I think this is the same root issue as in bug 1765416. (Upstream PyPi change causing a breakage in Poetry). The error message is slightly different, but I think it's closely related.

My changes up to and including D150814 fix this for me on Ubuntu and I can run ./mach vendor python successfully. For now, if you want to vendor something, the best way to do it is just to pull down those changes (ideally on Linux) and vendor on top of that stack. I'm afraid I can't land the stack until we upgrade the images on some CI machines, otherwise we'll have a ton of failures in CI.

I have at least one other issue on with Poetry on Windows (in my downstream venv lockfile stuff), so right now I'm leaning towards restricting ./mach vendor python to Linux only again. It used to be that way before, so I think it's reasonable to revert back to that. I do want to get the Windows Poetry problem(s) fixed, but it'll have to be upstream and I need to come up with a minimal test case first before I can report the issue.

Comment 4

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

(In reply to Andrew Halberstadt [:ahal] from comment #2)

Thanks to a tip from someone in the poetry issue, clearing the cache seemed to work:

$ poetry cache clear pypi --all

That is the right command to fix the upstream PyPi problem, but you also need to upgrade to the newer Poetry for it to pull data correctly, or it'll just cache garbage again anyway and still fail ./mach vendor python.

On Windows it's a bit trickier since if you don't have Poetry installed outside of the venv, that command will not work. When I was debugging this initially I had to manually go C:\Users\ahochheiden\AppData\Local\pypoetry and delete the cache myself.

Comment 5

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

(In reply to Andrew Halberstadt [:ahal] from comment #1)

Alex, fyi mach vendor is now broken. I'm still not 100% sure what's going on, but we may want to consider backing out your patch (I can confirm that upgrading to the parent of your revision "fixes" mach vendor python). Though this definitely isn't your fault or anything.. I'm just not sure how to fix this :).

It was broken before I landed that. See bug 1765416.

I will look into the looseversion problem though. I do not have those changes on the base of my Poetry upgrade stack, and if it doesn't have hashes that's definitely a problem.

Comment 6

[Andrew Halberstadt [:ahal]]

Thanks for all the context and patches! I can confirm that vendoring works if rebase on top of your stack :)

Comment 7

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Perfect. I think the looseversion problem you mentioned is just a symptom of the upstream PyPi change and older Poetry interacting, so no new problem there. I will do a proper rebase to that stack and push it so it's not annoying to deal with the merge conflicts for anyone else while this is in temporary limbo waiting for the Python3.7 bump.

Comment 8

[Mathew Hodson]

This should be Regressed by: 1777414

Comment 9

[Andrew Halberstadt [:ahal]]

Attachment: WIP: Bug 1782785 - Make vendor_python more robust when removing environment markers

Comment 10

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1782785 - Add "--clear-cache" command to `./mach vendor python` to enable clearing the pypi cache r?#build

Due to an upstream change on PyPi that makes the Poetry cache garbage,
this command is useful for after users upgrade to a newer version of
Poetry to remove the stale cached data, since in some cases it must be
done manually due to Poetry only being installed in a virtuelenv.

Comment 11

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1782785 - Make stripping environment markers more robost r?#build

Depends on D153661

Comment 12

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1782785 - Remove carriage returns from `./mach vendor python` output files to eliminate the differences from vendoring on Windows r?#build

The easiest way to do was to write the text to the file as bytes. This
makes "\n" actually a "LF", instead of a "CR LF" while in text mode,
keeping the output consistent with POSIX systems.

Depends on D153662

Comment 13

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

I started looking again into having Updatebot update python packages automatically but I'm blocked by this, so I'm adding it to the deps.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a91335e84926
Make stripping environment markers more robost r=ahal
https://hg.mozilla.org/integration/autoland/rev/294c49621bf8
Remove carriage returns from `./mach vendor python` output files to eliminate the differences from vendoring on Windows r=firefox-build-system-reviewers,glandium

Comment 16

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/a91335e84926
https://hg.mozilla.org/mozilla-central/rev/294c49621bf8

Comment 17

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

There are still more patches to land. This should stay open.

Comment 18

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1782785 - Patch poetry inside the virtualenv to fix the PyPi repository hash issue r?#build

There is a bug in Poetry 1.2.0a2 caused by a breaking change
on PyPi's end: https://github.com/pypi/warehouse/pull/11775
This bug was fixed in Poetry 1.2.0b3, but 1.2.0b1 dropped
support for Python 3.6, which is a requirement for us.
As a temporary workaround, we can patch the fix into our
virtualenv's copy of Poetry. This patch should be removed
once we switch to using a newer version of Poetry.

Depends on D161538

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c97e67ff2541
Patch poetry inside the virtualenv to fix the PyPi repository hash issue r=firefox-build-system-reviewers,glandium

Comment 21

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/c97e67ff2541

Comment 22

[Jim Mathies [:jimm]]

Can we close this out now?

Comment 23

[Sylvestre Ledru [:Sylvestre]]

Doesn't seem so. I just tried with a Debian testing

$ ./mach vendor python

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
poetry 1.2.0a2 requires packaging<21.0,>=20.4, but you have packaging 21.3 which is incompatible.
Successfully installed poetry-core-1.1.0a6
Resolving dependencies... (0.1s)<debug>PackageInfo:</debug> Invalid constraint (0.6 ; extra == 'rq') found in sentry-sdk-0.14.3 dependencies, skipping

Resolving dependencies... (0.5s)

Writing lock file
ERROR: Could not find a version that satisfies the requirement aiohttp==3.7.4.post0appdirs==1.4.4 (from versions: 0.1, 0.2, 0.3, 0.4, 0.4.1, 0.4.2, 0.4.3, 0.4.4, 0.5.0, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.14.2, 0.14.3, 0.14.4, 0.15.0, 0.15.1, 0.15.2, 0.15.3, 0.16.0, 0.16.1, 0.16.2, 0.16.3, 0.16.4, 0.16.5, 0.16.6, 0.17.0, 0.17.1, 0.17.2, 0.17.3, 0.17.4, 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.18.4, 0.19.0, 0.20.0, 0.20.1, 0.20.2, 0.21.0, 0.21.1, 0.21.2, 0.21.4, 0.21.5, 0.21.6, 0.22.0a0, 0.22.0b0, 0.22.0b1, 0.22.0b2, 0.22.0b3, 0.22.0b4, 0.22.0b5, 0.22.0b6, 0.22.0, 0.22.1, 0.22.2, 0.22.3, 0.22.4, 0.22.5, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.1.6, 1.2.0, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 2.0.0rc1, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6.post1, 2.0.7, 2.1.0, 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.2.5, 2.3.0a4, 2.3.0, 2.3.1, 2.3.2b2, 2.3.2b3, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.3.6, 2.3.7, 2.3.8, 2.3.9, 2.3.10, 3.0.0b0, 3.0.0b1, 3.0.0b2, 3.0.0b3, 3.0.0b4, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.0.5, 3.0.6, 3.0.7, 3.0.8, 3.0.9, 3.1.0, 3.1.1, 3.1.2, 3.1.3, 3.2.0, 3.2.1, 3.3.0a0, 3.3.0, 3.3.1, 3.3.2a0, 3.3.2, 3.4.0a0, 3.4.0a3, 3.4.0b1, 3.4.0b2, 3.4.0, 3.4.1, 3.4.2, 3.4.3, 3.4.4, 3.5.0a1, 3.5.0b1, 3.5.0b2, 3.5.0b3, 3.5.0, 3.5.1, 3.5.2, 3.5.3, 3.5.4, 3.6.0a0, 3.6.0a1, 3.6.0a2, 3.6.0a3, 3.6.0a4, 3.6.0a5, 3.6.0a6, 3.6.0a7, 3.6.0a8, 3.6.0a9, 3.6.0a11, 3.6.0a12, 3.6.0b0, 3.6.0, 3.6.1b3, 3.6.1b4, 3.6.1, 3.6.2a0, 3.6.2a1, 3.6.2a2, 3.6.2, 3.6.3, 3.7.0b0, 3.7.0b1, 3.7.0, 3.7.1, 3.7.2, 3.7.3, 3.7.4, 3.7.4.post0, 3.8.0a7, 3.8.0b0, 3.8.0, 3.8.1, 3.8.2, 3.8.3, 4.0.0a0, 4.0.0a1)
ERROR: No matching distribution found for aiohttp==3.7.4.post0appdirs==1.4.4
obsolete feature not enabled but 849 markers found!
Error running mach:

    ['vendor', 'python']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file vendor| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/home/sylvestre/dev/mozilla/mozilla-central.hg/obj-x86_64-pc-linux-gnu/_virtualenvs/vendor/bin/python', '-m', 'pip', 'download', '-r', '/tmp/tmpo90eq05x/requirements.no-markers.txt', '--no-deps', '--dest', '/tmp/tmp9n2cmkan', '--abi', 'none', '--platform', 'any']' returned non-zero exit status 1.

  File "/home/sylvestre/dev/mozilla/mozilla-central.hg/python/mozbuild/mozbuild/vendor/mach_commands.py", line 229, in vendor_python
    vendor_command.vendor(keep_extra_files)
  File "/home/sylvestre/dev/mozilla/mozilla-central.hg/python/mozbuild/mozbuild/vendor/vendor_python.py", line 54, in vendor
    subprocess.check_call(
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)

Comment 24

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

(In reply to Jim Mathies [:jimm] from comment #22)

Can we close this out now?

The current fix is just a temporary one. I want to keep this open until I can implement the permanent solution (Blocked until Python 3.7 on all CI machines).

(In reply to Sylvestre Ledru [:Sylvestre] from comment #23)

Doesn't seem so. I just tried with a Debian testing

I just tested on Debian 11.3 and it works fine for me. Can you try clearing your Poetry cache? (and pip cache if that doesn't work)

./mach python --virtualenv=vendor -m poetry cache clear pypi --all
pip3 cache purge

Comment 25

[Sylvestre Ledru [:Sylvestre]]

I just tested on Debian 11.3 and it works fine for me. Can you try clearing your Poetry cache? (and pip cache if that doesn't work)

./mach python --virtualenv=vendor -m poetry cache clear pypi --all
pip3 cache purge

\o/

It works.
Maybe we should suggest what you said in the error message?

./mach python --virtualenv=vendor -m poetry cache clear pypi --all
pip3 cache purge

Comment 26

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::Mach Core' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 27

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1782785 - Vendor `importlib_metadata` at `6.0.0` r?#build

This version is compatible with poetry 1.4.0 and removes blockers for various other ongoing projects.

https://github.com/python/importlib_metadata/blob/main/CHANGES.rst#v660

Depends on D149679

Comment 28

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1782785 - Update `poetry` to `1.4.0` r?#build

https://github.com/python-poetry/poetry/releases/tag/1.4.0https://github.com/python-poetry/poetry/releases/tag/1.4.0
https://github.com/python-poetry/poetry-core/releases/tag/1.5.1

The changes in third_party/python/requirements.txt, third_party/python/poetry.lock and mozbuild/test/vendor_requirements.txt are due to the newer version.

Depends on D150813

Comment 29

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1782785 - Removed the patching mechanism that was only needed for `poetry` version `1.2.0a2` r?#build

Depends on D171761

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dab5463ec959
Vendor `importlib_metadata` at `6.0.0` r=firefox-build-system-reviewers,glandium,ahal
https://hg.mozilla.org/integration/autoland/rev/4673371264b9
Update `poetry` to `1.4.0` r=ahal
https://hg.mozilla.org/integration/autoland/rev/4d64fa4277f8
Removed the patching mechanism that was only needed for `poetry` version `1.2.0a2` r=ahal

Comment 31

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/dab5463ec959
https://hg.mozilla.org/mozilla-central/rev/4673371264b9
https://hg.mozilla.org/mozilla-central/rev/4d64fa4277f8

Comment 32

[Sylvestre Ledru [:Sylvestre]]

Alex, can it be closed? thanks