Use pip v8's hash support instead of peep

RESOLVED FIXED

Status

Tree Management
Treeherder: Infrastructure
P3
normal
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: emorley, Assigned: emorley)

Tracking

Details

Attachments

(1 attachment)

(Assignee)

Description

3 years ago
The current Python buildpack uses pip, so doesn't make use of peep like we do on stage/prod.

We could use this custom buildpack that adds peep support:
https://github.com/pmclanahan/heroku-buildpack-python-peep

Or depending on timeline, perhaps official pip hashes support might have landed in pip by then, see:
https://github.com/pypa/pip/issues/1175 / https://github.com/pypa/pip/pull/3137
https://github.com/pypa/pip/issues/1035
https://github.com/pypa/pip/issues/468 / https://github.com/pypa/pip/pull/735
https://github.com/erikrose/peep/issues/7
(Assignee)

Updated

3 years ago
Summary: Consider using a fork of the Heroku Python buildpack that uses peep rather than pip → Use peep on Heroku instead of pip
(Assignee)

Comment 1

3 years ago
Fingers crossed https://github.com/pypa/pip/pull/3137 should be in pip v8, which once the python buildpack is updated means we can just use that instead :-)
(Assignee)

Comment 2

3 years ago
Pip v8 (due to be released in ~January) will have native support for hashes.
Summary: Use peep on Heroku instead of pip → Use pip v8's hash support on Heroku
(Assignee)

Updated

3 years ago
Depends on: 1230104
(Assignee)

Updated

3 years ago
Assignee: nobody → emorley
(Assignee)

Comment 3

3 years ago
I have WIP branches against Treeherder, heroku-buildpack-python and virtualenv for supporting pip v8, to expedite this. Once v8 is released I'll open some PRs.
(Assignee)

Comment 4

3 years ago
Pip v8 was released today.

virtualenv has been updated to include it in the new 14.0.0 release.

I've opened a PR against the Heroku Python buildpack, to update there too:
https://github.com/heroku/heroku-buildpack-python/pull/270
(Assignee)

Updated

3 years ago
Summary: Use pip v8's hash support on Heroku → Use pip v8's hash support instead of peep
(Assignee)

Updated

3 years ago
Depends on: 1241144
(Assignee)

Updated

3 years ago
Depends on: 1241519
(Assignee)

Updated

3 years ago
Depends on: 1241909
Created attachment 8711023 [details] [review]
[treeherder] mozilla:pip-require-hashes > mozilla:master
(Assignee)

Comment 6

3 years ago
Waiting on bug 1241519 and bug 1241909 before I ask for review.
Status: NEW → ASSIGNED
(Assignee)

Updated

2 years ago
Attachment #8711023 - Flags: review?(mdoglio)
Attachment #8711023 - Flags: review?(mdoglio) → review+

Comment 7

2 years ago
Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/a83f93029644ca370ef44c5d708ac02e7cd687d6
Bug 1213230 - Use pip 8's require-hashes feature instead of peep

As of pip 8, peep has now been integrated into pip.

Migrating from peep to this native feature has several advantages:
* It avoids the complexity/learning curve of using a wrapper around pip.
* It means we do not need to fork the official Heroku Python buildpack
  (which handles pip installation of requirements files) in order to use
  hash verification on Heroku. (Once the buildpack updates to pip 8.)
* Omitted sub-dependencies result in install-time errors rather than
  the user discovering omissions at run-time.
* pip's native caching is used, and all packages are installed in one
  pip invocation, so it's significantly faster.
* It has better handling of errors and corner cases.

Key facts about the native feature:
* hash-checking mode is enabled if at least one hash is found in the
  requirements files passed to pip, or can be force enabled by passing
  `--requires-hashes` when running `pip install`.
* Once enabled, hash-checking mode enforces that all packages:
  - are pinned to a specific version
  - have hashes listed
  - have all sub-dependencies specified
* Older versions of pip will error out if either `--require-hashes` or
  the requirements file `--hash` syntax is used, meaning it's not
  possible to accidentally lose hash-checking protection if the pip used
  is older than expected.

For more details, see:
https://pip.pypa.io/en/stable/user_guide/#hash-checking-mode
https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode

The pip version on Travis and in the Vagrant virtualenv has been updated
to 8.0.2 in bug 1241144, and the stage/prod virtualenv in bug 1241519.
The Heroku Python buildpack pip was updated in bug 1241909.

The requirements files hashes were ported using `peep port`, and then
comments/URLs re-added by hand.

https://github.com/mozilla/treeherder/commit/b2ecb99390185ed7883b2334c1b11855ddd0811b
Bug 1213230 - Remove peep.py

Since it's now unused.
(Assignee)

Comment 8

2 years ago
I've purged the virtualenvs and pre-updated the update.py scripts (to avoid the need to double deploy) for both stage and prod:

[emorley@treeherderadm.private.scl3 ~]$ ths
[emorley@treeherderadm.private.scl3 treeherder-service]$ sudo git fetch --quiet origin master && sudo git reset --hard FETCH_HEAD
HEAD is now at b2ecb99 Bug 1213230 - Remove peep.py
[emorley@treeherderadm.private.scl3 treeherder-service]$ cd ..
[emorley@treeherderadm.private.scl3 treeherder.allizom.org]$ ll
total 12
-rw-r--r--  1 treeherder treeherder 1290 Dec 29 15:00 treeherder-env.sh
drwxr-xr-x 14 treeherder treeherder 4096 Feb 11 18:34 treeherder-service
drwxr-xr-x  6 root       root       4096 Feb 11 18:33 venv
[emorley@treeherderadm.private.scl3 treeherder.allizom.org]$ sudo rm -r venv/

[emorley@treeherderadm.private.scl3 treeherder-service]$ thp
[emorley@treeherderadm.private.scl3 treeherder-service]$ sudo git fetch --quiet origin master && sudo git reset --hard FETCH_HEAD
HEAD is now at b2ecb99 Bug 1213230 - Remove peep.py
[emorley@treeherderadm.private.scl3 treeherder-service]$ cd ..
[emorley@treeherderadm.private.scl3 treeherder.mozilla.org]$ ll
total 12
-rw-r--r--  1 treeherder treeherder 1243 Feb  4 14:19 treeherder-env.sh
drwxr-xr-x 14 treeherder treeherder 4096 Feb  3 22:54 treeherder-service
drwxr-xr-x  6 root       root       4096 Feb  3 22:54 venv
[emorley@treeherderadm.private.scl3 treeherder.mozilla.org]$ sudo rm -r venv/
[emorley@treeherderadm.private.scl3 treeherder-service]$

And deployed to stage:
http://treeherderadm.private.scl3.mozilla.com/chief/treeherder.stage/logs/stage.1455540640
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Updated

2 years ago
Depends on: 1254961
You need to log in before you can comment on or make changes to this bug.