1920152 - Replace 'poetry' with 'uv' in `./mach vendor python'

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

uv now supports cross-platform lock files and is no longer experimental. It's much faster than poetry, and we can do a drop in replacement with little to no downsides. The only added constraint is that uv must be available via PyPi for the host platform to be able to run the vendor, which all tier 1 platforms are.

Comment 1

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1920152 - Replace 'poetry' with 'uv' for './mach vendor python' r?ahal

The vendoring steps are the same. Instead of using 'poetry' to handle
dependency resolution and creating a lock file, we now use 'uv'. That
lock file is then used to create a cross-platform 'requirements.txt'
file by 'uv'.

How the 'requirements.txt' file is consumed by 'pip' to download the
dependencies is unchanged.

Comment 2

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1920152 - Re-vendor using `./mach vendor python` (which now uses 'uv') r?ahal

The dependency resolution in 'poetry' was 'lowest', the default
dependency resolution in 'uv' is 'highest'. This means when we vendor,
we will vendor the newest/highest version possible of all transitive
dependencies that are universally/cross-platform compatible.

We could change from the default of 'highest', but I think this is
better. Various packages have been upgraded, and some have been
removed/replaced by others, and that's why there's various changes in
the <site>.txt files.

Comment 3

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1920152 - Add various additional features to `./mach vendor python` r?ahal

• Added argument "--add", which allows a user to specify a dependency to
  add via the command line, rather than editing the 'pyproject.toml'
  directly.
• Added argument "--remove", which is the opposite of add, with slightly
  different syntax (This does not require the "==<major.minor.patch>"
  portion of a dependency.
  (The above arguments are both features of uv, and we're essentially just
  passing them through from Mach to uv)
• Added a hash of the lockfile, so that if you run './mach vendor
  python', but the lockfile has not changed, there's nothing to do, so the
  rest of vendoring is skipped.
• Added "--force" to force a vendor, which essentially just skips the
  lockfile hashing steps, and always forces a vendor, regardless of the
  change status.

Comment 4

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1920152 - Implement upgrading vendored dependencies via uv r?ahal

Currently, running running ./mach vendor python --upgrade will not do
anything different, because all dependencies are pinned. In later
patches, we can unpin dependencies and that will make any future runs
with --upgrade added pull the newest versions from PyPi that are
universally compatible.

Comment 5

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1920152 - Doc updates for `./mach vendor python` r?ahal

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2bfb683e59cb
Replace 'poetry' with 'uv' for './mach vendor python' r=ahal,mach-reviewers
https://hg.mozilla.org/integration/autoland/rev/1cbd01154dae
Re-vendor using `./mach vendor python` (which now uses 'uv') r=ahal,mach-reviewers
https://hg.mozilla.org/integration/autoland/rev/00822681fcea
Add various additional features to `./mach vendor python` r=ahal,mach-reviewers
https://hg.mozilla.org/integration/autoland/rev/ce1186bb2063
Implement upgrading vendored dependencies via uv r=ahal
https://hg.mozilla.org/integration/autoland/rev/d694b46ede60
Doc updates for `./mach vendor python` r=ahal

Comment 7

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1920152 - Prevent the absolute path from being written in `requirements.txt` and re-vendor r?ahal

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1c7d5f288462
Prevent the absolute path from being written in `requirements.txt` and re-vendor r=ahal,mach-reviewers

Comment 9

[tszentpeteri]

Backed out for causing gecko bustages related to post-signing-dummy-nightly-1.

• Backout link
• Push with failures
• Failure Log:

[task 2024-10-03T17:32:09.398Z] Generated 432 tasks for kind build-components
[task 2024-10-03T17:32:09.398Z] Loading tasks for kind signing
[task 2024-10-03T17:32:09.402Z] Generating tasks for signing sign-build
[task 2024-10-03T17:32:09.545Z] Generated 315 tasks for kind signing
[task 2024-10-03T17:32:09.545Z] Loading tasks for kind post-signing-dummy
[task 2024-10-03T17:32:09.550Z] Generating tasks for post-signing-dummy post-signing-dummy
[task 2024-10-03T17:32:09.551Z] Error loading tasks for kind post-signing-dummy:
[task 2024-10-03T17:32:09.551Z] Traceback (most recent call last):
[task 2024-10-03T17:32:09.551Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 311, in _run
[task 2024-10-03T17:32:09.551Z]     new_tasks = kind.load_tasks(
[task 2024-10-03T17:32:09.551Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 76, in load_tasks
[task 2024-10-03T17:32:09.551Z]     tasks = [
[task 2024-10-03T17:32:09.551Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 76, in <listcomp>
[task 2024-10-03T17:32:09.551Z]     tasks = [
[task 2024-10-03T17:32:09.551Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 2311, in check_run_task_caches
[task 2024-10-03T17:32:09.551Z]     for task in tasks:
[task 2024-10-03T17:32:09.551Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 2223, in check_task_dependencies
[task 2024-10-03T17:32:09.551Z]     raise Exception(
[task 2024-10-03T17:32:09.551Z] Exception: task post-signing-dummy/post-signing-dummy-nightly-1 has too many dependencies (105 > 99)
[task 2024-10-03T17:32:09.555Z] Traceback (most recent call last):
[task 2024-10-03T17:32:09.555Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 209, in taskgraph_decision
[task 2024-10-03T17:32:09.555Z]     ret = taskgraph_commands["decision"].func(options)
[task 2024-10-03T17:32:09.555Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/main.py", line 521, in decision
[task 2024-10-03T17:32:09.555Z]     taskgraph_decision(options)
[task 2024-10-03T17:32:09.555Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/decision.py", line 202, in taskgraph_decision
[task 2024-10-03T17:32:09.555Z]     full_task_json = tgg.full_task_graph.to_json()
[task 2024-10-03T17:32:09.555Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 170, in full_task_graph
[task 2024-10-03T17:32:09.555Z]     return self._run_until("full_task_graph")
[task 2024-10-03T17:32:09.555Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 430, in _run_until
[task 2024-10-03T17:32:09.555Z]     k, v = next(self._run)  # type: ignore
[task 2024-10-03T17:32:09.555Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 311, in _run
[task 2024-10-03T17:32:09.555Z]     new_tasks = kind.load_tasks(
[task 2024-10-03T17:32:09.555Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 76, in load_tasks
[task 2024-10-03T17:32:09.555Z]     tasks = [
[task 2024-10-03T17:32:09.555Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 76, in <listcomp>
[task 2024-10-03T17:32:09.555Z]     tasks = [
[task 2024-10-03T17:32:09.555Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 2311, in check_run_task_caches
[task 2024-10-03T17:32:09.555Z]     for task in tasks:
[task 2024-10-03T17:32:09.555Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/transforms/task.py", line 2223, in check_task_dependencies
[task 2024-10-03T17:32:09.555Z]     raise Exception(
[task 2024-10-03T17:32:09.555Z] Exception: task post-signing-dummy/post-signing-dummy-nightly-1 has too many dependencies (105 > 99)
[task 2024-10-03T17:32:09.555Z] Creating default state directory: /builds/worker/.mozbuild
[task 2024-10-03T17:32:09.555Z] Creating local state directory: /builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69
[taskcluster 2024-10-03 17:32:10.257Z] === Task Finished ===
[taskcluster 2024-10-03 17:32:21.650Z] Unsuccessful task run with exit code: 1 completed in 45.912 seconds

Comment 10

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

:ahal landed my patch with his, and this was caused by his. He'll take care of it in bug 1922536.

Comment 11

[tszentpeteri]

https://hg.mozilla.org/mozilla-central/rev/2bfb683e59cb
https://hg.mozilla.org/mozilla-central/rev/1cbd01154dae
https://hg.mozilla.org/mozilla-central/rev/00822681fcea
https://hg.mozilla.org/mozilla-central/rev/ce1186bb2063
https://hg.mozilla.org/mozilla-central/rev/d694b46ede60

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1bc8705b3a42
Prevent the absolute path from being written in `requirements.txt` and re-vendor r=ahal,mach-reviewers