Move existing PyPI installations to use centralized dependency management
Categories
(Firefox Build System :: Mach Core, enhancement, P2)
Tracking
(Not tracked)
People
(Reporter: mhentges, Unassigned)
References
(Blocks 1 open bug)
Details
(Keywords: leave-open)
Attachments
(5 files, 2 obsolete files)
Benefits:
- Faster: we skip pip's attempted installation of all the packages
- More stable: virtualenvs are no longer mutated with ad-hoc, potentially incompatible packages
Reporter | ||
Updated•3 years ago
|
Reporter | ||
Updated•3 years ago
|
Reporter | ||
Comment 1•3 years ago
|
||
When porting commands to have their own virtualenv,
it's useful to have helpful warnings when the associated
requirements definition file is missing.
Depends on D122892
Reporter | ||
Comment 2•3 years ago
|
||
This allows strategic grouping of dependencies, which can be convenient.
Depends on D122894
Reporter | ||
Comment 3•3 years ago
|
||
This should improve the workflow when sharding off commands into their
own virtualenvs.
Depends on D122895
Updated•3 years ago
|
Reporter | ||
Comment 4•3 years ago
|
||
Creates/updates virtualenvs for four different mach commands, replacing
their ad-hoc usage of install_pip_package()
.
Depends on D122900
Reporter | ||
Comment 5•3 years ago
|
||
The majority of the usages that we should be porting to the centralized system are:
- _run_pip()
- ✅ Vendoring
- mozperftest.utils.install_package()
- mozperfest...perfboard.influx
- install_package("influxdb")
- install_package("grafana_api")
- mozperftest...system.pingserver install_package("edgeping", "requests")
- mozperftest...system.proxy install_package("mozinfo", "mozlog", "mozproxy")
- mozperftest...browsertime.runner.setup() install_package(get_dependencies())
- mozperftest.mach_commands._run_tests() install_package("coverage", "attrs")
- mozproxy.tests.test_command_line.install_mozproxy() install_package("mozinfo", "mozlog", "mozproxy")
- performance.hooks_android_view..before_iterations() install_package("gitpython")
- mozperfest...perfboard.influx
- python.mach_commands.python() run_pip(["install", "ipython"]) => [centralize]
- install_pip_package()
- ipython
- upload_generated_sources boto3==1.4.4
- Vendoring pip_tools (just do
python -m piptools compile ...
) - PypiBasedTool
- logspam
- mozregression
- mozilla-log-spam
=> Use virtualenv instead
- install_pip_requirements()
- ./mach python --requirements
- run_python_tests test["requirements"]
- condprofile/requirements/base.txt
- websocketprocessbridge_requirements_3.txt
- wpt requirements:
- AndroidWebview (requirements_android_webview.txt)
- ChromeAndroid (requirements_chrome_android.txt)
- etc
=> Maybe have "create venv for requirements file" ad-hoc thinger
- Manual "sys.executable -m pip install" shenanigans
Reporter | ||
Comment 6•3 years ago
|
||
We've been installing piptools
so that we can use the pip-compile
binary. However, we can just as easily do python -m piptools compile ...
instead, which also allows us to skip a redundant install.
Depends on D123124
Reporter | ||
Comment 7•3 years ago
|
||
Now that piptools
is no longer installed from the vendored location,
there are no remaining cases where we are doing install_pip_package()
with a vendored package.
Depends on D123240
Pushed by mhentges@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/bcd2acfba0e7 Use pip-compile directly from vendored location r=ahal https://hg.mozilla.org/integration/autoland/rev/792f5edc9534 Remove `vendor` argument from `install_pip_package()` r=ahal
Pushed by mhentges@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/3df9bc9d1f4f Allow empty lines in python requirements files r=ahal
Comment 10•3 years ago
|
||
bugherder |
Comment 11•3 years ago
|
||
Pushed by mhentges@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/118c4908bfaa Warn if virtualenv req definition is missing r=ahal
Comment 12•3 years ago
|
||
bugherder |
Updated•3 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Comment 13•2 years ago
|
||
Pushed by mhentges@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/09af1ea3e6e3 Move low-hanging fruit commands to centralized Python dep system r=ahal
Comment 14•2 years ago
|
||
bugherder |
Reporter | ||
Updated•2 years ago
|
Comment 15•2 years ago
|
||
The leave-open keyword is there and there is no activity for 6 months.
:ahochheiden, maybe it's time to close this bug?
For more information, please visit auto_nag documentation.
Comment hidden (spam) |
Comment hidden (collapsed) |
Description
•