1723237 - Move existing PyPI installations to use centralized dependency management

Status: NEW

(Firefox Build System :: Mach Core, enhancement, P2)

Description

[Mitchell Hentges [:mhentges] 🦀]

Benefits:

• Faster: we skip pip's attempted installation of all the packages
• More stable: virtualenvs are no longer mutated with ad-hoc, potentially incompatible packages

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1723237: Warn if virtualenv req definition is missing

When porting commands to have their own virtualenv,
it's useful to have helpful warnings when the associated
requirements definition file is missing.

Depends on D122892

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1723237: Allow empty lines in python requirements files

This allows strategic grouping of dependencies, which can be convenient.

Depends on D122894

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1723237: Improve warning when requirements file is missing

This should improve the workflow when sharding off commands into their
own virtualenvs.

Depends on D122895

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1723237: Move low-hanging fruit commands to centralized Python dep system

Creates/updates virtualenvs for four different mach commands, replacing
their ad-hoc usage of install_pip_package().

Depends on D122900

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

The majority of the usages that we should be porting to the centralized system are:

• _run_pip()
  • ✅ Vendoring
  • mozperftest.utils.install_package()
    • mozperfest...perfboard.influx
      • install_package("influxdb")
      • install_package("grafana_api")
    • mozperftest...system.pingserver install_package("edgeping", "requests")
    • mozperftest...system.proxy install_package("mozinfo", "mozlog", "mozproxy")
    • mozperftest...browsertime.runner.setup() install_package(get_dependencies())
    • mozperftest.mach_commands._run_tests() install_package("coverage", "attrs")
    • mozproxy.tests.test_command_line.install_mozproxy() install_package("mozinfo", "mozlog", "mozproxy")
    • performance.hooks_android_view..before_iterations() install_package("gitpython")
  • python.mach_commands.python() run_pip(["install", "ipython"]) => [centralize]
• install_pip_package()
  • ipython
  • upload_generated_sources boto3==1.4.4
  • Vendoring pip_tools (just do python -m piptools compile ...)
  • PypiBasedTool
    • logspam
    • mozregression
    • mozilla-log-spam
      => Use virtualenv instead
• install_pip_requirements()
  • ./mach python --requirements
  • run_python_tests test["requirements"]
  • condprofile/requirements/base.txt
  • websocketprocessbridge_requirements_3.txt
  • wpt requirements:
    • AndroidWebview (requirements_android_webview.txt)
    • ChromeAndroid (requirements_chrome_android.txt)
    • etc
      => Maybe have "create venv for requirements file" ad-hoc thinger
• Manual "sys.executable -m pip install" shenanigans

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1723237: Use pip-compile directly from vendored location

We've been installing piptools so that we can use the pip-compile
binary. However, we can just as easily do python -m piptools compile ... instead, which also allows us to skip a redundant install.

Depends on D123124

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1723237: Remove `vendor` argument from `install_pip_package()`

Now that piptools is no longer installed from the vendored location,
there are no remaining cases where we are doing install_pip_package()
with a vendored package.

Depends on D123240

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bcd2acfba0e7
Use pip-compile directly from vendored location r=ahal
https://hg.mozilla.org/integration/autoland/rev/792f5edc9534
Remove `vendor` argument from `install_pip_package()` r=ahal

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3df9bc9d1f4f
Allow empty lines in python requirements files r=ahal

Comment 10

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/bcd2acfba0e7
https://hg.mozilla.org/mozilla-central/rev/792f5edc9534
https://hg.mozilla.org/mozilla-central/rev/3df9bc9d1f4f

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/118c4908bfaa
Warn if virtualenv req definition is missing r=ahal

Comment 12

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/118c4908bfaa

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/09af1ea3e6e3
Move low-hanging fruit commands to centralized Python dep system r=ahal

Comment 14

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/09af1ea3e6e3

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:ahochheiden, maybe it's time to close this bug?
For more information, please visit auto_nag documentation.

Comment 16

[Alex Hochheiden [:ahochheiden] [he/him]]

There's still work to be done here.

Comment 18

[David Lawrence [:dkl]]

The content of attachment 9329146 [details] has been deleted for the following reason:

Spam