1681560 - Ad-hoc installation of pip dependencies should include SHA

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement, P3)

Description

[Mitchell Hentges [:mhentges] 🦀]

When installing native packages (e.g.: glean, psutil, zstandard), we just identify them by version. We should include a SHA hash as well.

Comment 1

[Connor Sheehan [:sheehan]]

At the moment, installing native packages is done by calling VirtualenvManager.install_pip_package, which calls out to pip install under the hood, passing the argument directly to the CLI as a requirements specifier. Unfortunately pip install doesn't support passing hashes on the command line (see https://github.com/pypa/pip/issues/3257, though it seems abandoned).

We'll need to add the dependencies to a requirements file if we want to pin the SHAs. We could add all the SHAs to a single file (native_requirements.txt or something like that) but this would mean that when you try to install one of the packages, you always install all three. We could also pin the specific version of each package as it's own requirements file, then do a pip install -r zstandard_requirement.txt or similar to install.

One more thing I've noticed is that we don't pin a specific version of some of the packages. For example we install zstandard using the requirements specifier zstandard>=0.9.0,<=0.13.0. We'll need to settle on a specific version if we want to pin hashes of these packages.

Mitch, do you know if there's a reason we don't pin a specific version of zstandard? We also don't use a set version for glean_sdk. I dug around the VCS annotate but didn't see any comments about why the specific package versions were used.

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

We could also pin the specific version of each package as it's own requirements file, then do a pip install -r zstandard_requirement.txt or similar to install.

True. Alternatively, in the same way as ./mach vendor works, we can generate a temporary requirements.txt file (except with a single package and SHA), install it, then remove the file.

One more thing I've noticed is that we don't pin a specific version of some of the packages. For example we install zstandard using the requirements specifier zstandard>=0.9.0,<=0.13.0

Good question, that's been a flexible requirement specifier for as long as I could tell.
I dug backwards through the commit history and found this patch diff where it was added. Perhaps we have zstandard==0.9.0 in a CI package mirror?
You can pin this to 0.13.0 (which developers should have locally) and run a CI job that depends on it (such as this one) to see if CI will be happy.

We also don't use a set version for glean_sdk

Bam.

---

Hope that helps :)

Comment 3

[Connor Sheehan [:sheehan]]

Attachment: Bug 1681560: install native packages using requirements files with pinned hashes r?#build

At the moment installing Python packages with native code is done by calling
pip install <package> and does not enforce any SHA hash for installed
dependencies, nor does it enforce a specific version to be installed.

This commit adds requirements.in and requirements.txt files for native
packages and changes these packages to be installed by running pip install
and passing the requirements file for the package. This allows us to pin the
SHA of the various dependencies. The .txt files are generated using
pip-compile.

We also add the new requirements files to the sparse profile for mach.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a31a95d6a390
install native packages using requirements files with pinned hashes r=firefox-build-system-reviewers,mhentges

Comment 5

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/a31a95d6a390