Open Bug 1723237 Opened 3 months ago Updated 15 days ago

Move existing PyPI installations to use centralized dependency management

Categories

(Firefox Build System :: Mach Core, enhancement, P2)

enhancement

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: mhentges, Assigned: mhentges)

References

(Blocks 1 open bug)

Details

(Keywords: leave-open)

Attachments

(5 files, 1 obsolete file)

Benefits:

  • Faster: we skip pip's attempted installation of all the packages
  • More stable: virtualenvs are no longer mutated with ad-hoc, potentially incompatible packages
Blocks: 1712131
Keywords: leave-open
Priority: -- → P2

When porting commands to have their own virtualenv,
it's useful to have helpful warnings when the associated
requirements definition file is missing.

Depends on D122892

This allows strategic grouping of dependencies, which can be convenient.

Depends on D122894

This should improve the workflow when sharding off commands into their
own virtualenvs.

Depends on D122895

Assignee: nobody → mhentges
Status: NEW → ASSIGNED

Creates/updates virtualenvs for four different mach commands, replacing
their ad-hoc usage of install_pip_package().

Depends on D122900

The majority of the usages that we should be porting to the centralized system are:

  • _run_pip()
    • ✅ Vendoring
    • mozperftest.utils.install_package()
      • mozperfest...perfboard.influx
        • install_package("influxdb")
        • install_package("grafana_api")
      • mozperftest...system.pingserver install_package("edgeping", "requests")
      • mozperftest...system.proxy install_package("mozinfo", "mozlog", "mozproxy")
      • mozperftest...browsertime.runner.setup() install_package(get_dependencies())
      • mozperftest.mach_commands._run_tests() install_package("coverage", "attrs")
      • mozproxy.tests.test_command_line.install_mozproxy() install_package("mozinfo", "mozlog", "mozproxy")
      • performance.hooks_android_view..before_iterations() install_package("gitpython")
    • python.mach_commands.python() run_pip(["install", "ipython"]) => [centralize]
  • install_pip_package()
    • ipython
    • upload_generated_sources boto3==1.4.4
    • Vendoring pip_tools (just do python -m piptools compile ...)
    • PypiBasedTool
      • logspam
      • mozregression
      • mozilla-log-spam
        => Use virtualenv instead
  • install_pip_requirements()
    • ./mach python --requirements
    • run_python_tests test["requirements"]
    • condprofile/requirements/base.txt
    • websocketprocessbridge_requirements_3.txt
    • wpt requirements:
      • AndroidWebview (requirements_android_webview.txt)
      • ChromeAndroid (requirements_chrome_android.txt)
      • etc
        => Maybe have "create venv for requirements file" ad-hoc thinger
  • Manual "sys.executable -m pip install" shenanigans

We've been installing piptools so that we can use the pip-compile
binary. However, we can just as easily do python -m piptools compile ... instead, which also allows us to skip a redundant install.

Depends on D123124

Now that piptools is no longer installed from the vendored location,
there are no remaining cases where we are doing install_pip_package()
with a vendored package.

Depends on D123240

Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/bcd2acfba0e7
Use pip-compile directly from vendored location r=ahal
https://hg.mozilla.org/integration/autoland/rev/792f5edc9534
Remove `vendor` argument from `install_pip_package()` r=ahal
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3df9bc9d1f4f
Allow empty lines in python requirements files r=ahal
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/118c4908bfaa
Warn if virtualenv req definition is missing r=ahal
Attachment #9236710 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.