1213230 - Use pip v8's hash support instead of peep

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P3)

Description

[Ed Morley [:emorley]]

The current Python buildpack uses pip, so doesn't make use of peep like we do on stage/prod.

We could use this custom buildpack that adds peep support:
https://github.com/pmclanahan/heroku-buildpack-python-peep

Or depending on timeline, perhaps official pip hashes support might have landed in pip by then, see:
https://github.com/pypa/pip/issues/1175 / https://github.com/pypa/pip/pull/3137
https://github.com/pypa/pip/issues/1035
https://github.com/pypa/pip/issues/468 / https://github.com/pypa/pip/pull/735
https://github.com/erikrose/peep/issues/7

Comment 1

[Ed Morley [:emorley]]

Fingers crossed https://github.com/pypa/pip/pull/3137 should be in pip v8, which once the python buildpack is updated means we can just use that instead :-)

Comment 2

[Ed Morley [:emorley]]

Pip v8 (due to be released in ~January) will have native support for hashes.

Comment 3

[Ed Morley [:emorley]]

I have WIP branches against Treeherder, heroku-buildpack-python and virtualenv for supporting pip v8, to expedite this. Once v8 is released I'll open some PRs.

Comment 4

[Ed Morley [:emorley]]

Pip v8 was released today.

virtualenv has been updated to include it in the new 14.0.0 release.

I've opened a PR against the Heroku Python buildpack, to update there too:
https://github.com/heroku/heroku-buildpack-python/pull/270

Comment 5

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:pip-require-hashes > mozilla:master

Comment 6

[Ed Morley [:emorley]]

Waiting on bug 1241519 and bug 1241909 before I ask for review.

Comment 7

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/a83f93029644ca370ef44c5d708ac02e7cd687d6
Bug 1213230 - Use pip 8's require-hashes feature instead of peep

As of pip 8, peep has now been integrated into pip.

Migrating from peep to this native feature has several advantages:
* It avoids the complexity/learning curve of using a wrapper around pip.
* It means we do not need to fork the official Heroku Python buildpack
  (which handles pip installation of requirements files) in order to use
  hash verification on Heroku. (Once the buildpack updates to pip 8.)
* Omitted sub-dependencies result in install-time errors rather than
  the user discovering omissions at run-time.
* pip's native caching is used, and all packages are installed in one
  pip invocation, so it's significantly faster.
* It has better handling of errors and corner cases.

Key facts about the native feature:
* hash-checking mode is enabled if at least one hash is found in the
  requirements files passed to pip, or can be force enabled by passing
  `--requires-hashes` when running `pip install`.
* Once enabled, hash-checking mode enforces that all packages:
  - are pinned to a specific version
  - have hashes listed
  - have all sub-dependencies specified
* Older versions of pip will error out if either `--require-hashes` or
  the requirements file `--hash` syntax is used, meaning it's not
  possible to accidentally lose hash-checking protection if the pip used
  is older than expected.

For more details, see:
https://pip.pypa.io/en/stable/user_guide/#hash-checking-mode
https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode

The pip version on Travis and in the Vagrant virtualenv has been updated
to 8.0.2 in bug 1241144, and the stage/prod virtualenv in bug 1241519.
The Heroku Python buildpack pip was updated in bug 1241909.

The requirements files hashes were ported using `peep port`, and then
comments/URLs re-added by hand.

https://github.com/mozilla/treeherder/commit/b2ecb99390185ed7883b2334c1b11855ddd0811b
Bug 1213230 - Remove peep.py

Since it's now unused.

Comment 8

[Ed Morley [:emorley]]

I've purged the virtualenvs and pre-updated the update.py scripts (to avoid the need to double deploy) for both stage and prod:

[[e-mail]3 ~]$ ths
[[e-mail]3 treeherder-service]$ sudo git fetch --quiet origin master && sudo git reset --hard FETCH_HEAD
HEAD is now at b2ecb99 Bug 1213230 - Remove peep.py
[[e-mail]3 treeherder-service]$ cd ..
[[e-mail]3 treeherder.allizom.org]$ ll
total 12
-rw-r--r--  1 treeherder treeherder 1290 Dec 29 15:00 treeherder-env.sh
drwxr-xr-x 14 treeherder treeherder 4096 Feb 11 18:34 treeherder-service
drwxr-xr-x  6 root       root       4096 Feb 11 18:33 venv
[[e-mail]3 treeherder.allizom.org]$ sudo rm -r venv/

[[e-mail]3 treeherder-service]$ thp
[[e-mail]3 treeherder-service]$ sudo git fetch --quiet origin master && sudo git reset --hard FETCH_HEAD
HEAD is now at b2ecb99 Bug 1213230 - Remove peep.py
[[e-mail]3 treeherder-service]$ cd ..
[[e-mail]3 treeherder.mozilla.org]$ ll
total 12
-rw-r--r--  1 treeherder treeherder 1243 Feb  4 14:19 treeherder-env.sh
drwxr-xr-x 14 treeherder treeherder 4096 Feb  3 22:54 treeherder-service
drwxr-xr-x  6 root       root       4096 Feb  3 22:54 venv
[[e-mail]3 treeherder.mozilla.org]$ sudo rm -r venv/
[[e-mail]3 treeherder-service]$

And deployed to stage:
http://treeherderadm.private.scl3.mozilla.com/chief/treeherder.stage/logs/stage.1455540640