Open Bug 1723237 Opened 3 months ago Updated 15 days ago

Move existing PyPI installations to use centralized dependency management


(Firefox Build System :: Mach Core, enhancement, P2)



(Not tracked)



(Reporter: mhentges, Assigned: mhentges)


(Blocks 1 open bug)


(Keywords: leave-open)


(5 files, 1 obsolete file)


  • Faster: we skip pip's attempted installation of all the packages
  • More stable: virtualenvs are no longer mutated with ad-hoc, potentially incompatible packages
Blocks: 1712131
Keywords: leave-open
Priority: -- → P2

When porting commands to have their own virtualenv,
it's useful to have helpful warnings when the associated
requirements definition file is missing.

Depends on D122892

This allows strategic grouping of dependencies, which can be convenient.

Depends on D122894

This should improve the workflow when sharding off commands into their
own virtualenvs.

Depends on D122895

Assignee: nobody → mhentges

Creates/updates virtualenvs for four different mach commands, replacing
their ad-hoc usage of install_pip_package().

Depends on D122900

The majority of the usages that we should be porting to the centralized system are:

  • _run_pip()
    • ✅ Vendoring
    • mozperftest.utils.install_package()
      • mozperfest...perfboard.influx
        • install_package("influxdb")
        • install_package("grafana_api")
      • mozperftest...system.pingserver install_package("edgeping", "requests")
      • mozperftest...system.proxy install_package("mozinfo", "mozlog", "mozproxy")
      • mozperftest...browsertime.runner.setup() install_package(get_dependencies())
      • mozperftest.mach_commands._run_tests() install_package("coverage", "attrs")
      • mozproxy.tests.test_command_line.install_mozproxy() install_package("mozinfo", "mozlog", "mozproxy")
      • performance.hooks_android_view..before_iterations() install_package("gitpython")
    • python.mach_commands.python() run_pip(["install", "ipython"]) => [centralize]
  • install_pip_package()
    • ipython
    • upload_generated_sources boto3==1.4.4
    • Vendoring pip_tools (just do python -m piptools compile ...)
    • PypiBasedTool
      • logspam
      • mozregression
      • mozilla-log-spam
        => Use virtualenv instead
  • install_pip_requirements()
    • ./mach python --requirements
    • run_python_tests test["requirements"]
    • condprofile/requirements/base.txt
    • websocketprocessbridge_requirements_3.txt
    • wpt requirements:
      • AndroidWebview (requirements_android_webview.txt)
      • ChromeAndroid (requirements_chrome_android.txt)
      • etc
        => Maybe have "create venv for requirements file" ad-hoc thinger
  • Manual "sys.executable -m pip install" shenanigans

We've been installing piptools so that we can use the pip-compile
binary. However, we can just as easily do python -m piptools compile ... instead, which also allows us to skip a redundant install.

Depends on D123124

Now that piptools is no longer installed from the vendored location,
there are no remaining cases where we are doing install_pip_package()
with a vendored package.

Depends on D123240

Pushed by
Use pip-compile directly from vendored location r=ahal
Remove `vendor` argument from `install_pip_package()` r=ahal
Pushed by
Allow empty lines in python requirements files r=ahal
Pushed by
Warn if virtualenv req definition is missing r=ahal
Attachment #9236710 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.