1598482 - Missing Taskcluster third_party dependencies make 'mach vendor python' hard to use

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P3)

Description

[Māris Fogels [:mars] (please needinfo)]

The taskcluster dependency on aiohttp and it's sub-dependencies are missing from the third_party/python directory. The missing deps show up in the "to be upgraded" list whenever someone attempts to use mach vendor python to upgrade an individual package. It makes use of mach vendor python confusing and tedious to repair.

Steps to reproduce:

1. mach vendor python taskcluster==6.0.0
2. hg st third_party to see the list of missing packages the vendor command has added.

Expected output:
No changes to the third_party/python/ directory.

Actual output:
aiohttp, yarl and friends have appeared in the third_party/python/ directory.

To see how we know taskcluster is causing the problem, run:

1. cd obj-*/_virtualenvs/init_py3/bin
2. ./pip-compile ../../../../third_party/python/requirements.in --output-file=-

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

It sounds like the fix would be to install those dependencies?

Comment 2

[Andrew Halberstadt [:ahal]]

Yes, we either need to vendor the dependencies, or stop vendoring the taskcluster client and install it at runtime as needed (only appears used by mozrelease).

Comment 3

[Ricky Stewart]

FWIW this issue has gotten worse and doing no-op "mach vendor python" runs now installs even more packages.

I wonder if a feasible solution is to "declare bankruptcy" by just vendoring all these packages once and for all in a giant patch.

Comment 4

[Māris Fogels [:mars] (please needinfo)]

I'm +1 for vendoring the missing libraries in a single patch. I'm up for reviewing the patch for whoever does this.

Comment 5

[Axel Hecht [:Pike]]

Bug 1601140 fixed this, but I don't know how to properly resolve this bug. Dupe, worksforme, depend, morph into "vendor this again once it's py3" and unblock mach-busted ... ? Any preferences?

Comment 6

[Mark Banner (:standard8)]

Marking as fixed by bug 1601140, based on comment 5.