1725708 - Include all possible vendored dependencies under "requirements.txt"

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, task)

Description

[Mitchell Hentges [:mhentges] 🦀]

A fair amount of the packages in third_party/python/ have been copied in there manually, causing two downsides:

• They aren't asserted to be compatible with each other
• It's not obvious what some of the versions of the packages are

We should port them to use the centralized system.

Comment 1

[Andrew Halberstadt [:ahal]]

You can probably also remove the warning while you're here.

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725708: Remove unused "pep487" vendored package

There's no usages of the pep487 module.

Depends on D123120

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725708: Move all possible vendored deps to centralized system

Note that, as part of adding this packages to the automated vendoring
system, some dependencies were automatically added - most notably,
dependencies of taskcluster that become visible with Python 3.6+.

Adds **/.git to the exclusions because:

• .git is part of our .hgignore, but
• .git is part of the aiohttp tar.gz file.

Since the file isn't needed for pip install-ing aiohttp,
and since we want ./mach vendor python to be a no-op when there's
no requirement changes, we exclude it.

Depends on D123121

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725708: [SQUASH] Actual `./mach vendor python` changes

Depends on D123122

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725708: `./mach vendor python` should populate from a clean state

Rather than deleting the expected target directory of each package
that's being vendored, clear the whole third_party/python directory
and re-populate it from scratch.

As part of this, there's an "exclusion" list for packages that can't
be vendored from PyPI.

This has some benefits:

• It'll be harder to forget scraps of files and directories and leave
  them in third_party/python.
• The exclusion list makes it more clear which packages are managed
  manually, and the friction it adds to the workflow will guide
  developers to use "requirements.in" instead.

The test_up_to_date_vendor test will verify that the vendor directory
is always clean.

Depends on D123123

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/11de3a4312bd
Remove unused "pep487" vendored package r=ahal
https://hg.mozilla.org/integration/autoland/rev/75336eb10063
Move all possible vendored deps to centralized system r=ahal
https://hg.mozilla.org/integration/autoland/rev/90b9ff20b311
`./mach vendor python` should populate from a clean state r=ahal

Comment 7

[Cosmin Sabou [:CosminS]]

Backed out for causing invalid handle xpcshell failures.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=U48eqelWQge_ZBWzSGPX-A.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cusercancel&revision=90b9ff20b311609163c786cdee830ff9a132d575

Failure log: https://treeherder.mozilla.org/logviewer?job_id=350477627&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/6b58b83b626e01f7b2af85156c293c9bccd82816

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/746509f3a640
Remove unused "pep487" vendored package r=ahal
https://hg.mozilla.org/integration/autoland/rev/06115f01c86f
Move all possible vendored deps to centralized system r=ahal
https://hg.mozilla.org/integration/autoland/rev/dd841f22c83e
`./mach vendor python` should populate from a clean state r=ahal

Comment 9

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/746509f3a640
https://hg.mozilla.org/mozilla-central/rev/06115f01c86f
https://hg.mozilla.org/mozilla-central/rev/dd841f22c83e