The current Python buildpack uses pip, so doesn't make use of peep like we do on stage/prod. We could use this custom buildpack that adds peep support: https://github.com/pmclanahan/heroku-buildpack-python-peep Or depending on timeline, perhaps official pip hashes support might have landed in pip by then, see: https://github.com/pypa/pip/issues/1175 / https://github.com/pypa/pip/pull/3137 https://github.com/pypa/pip/issues/1035 https://github.com/pypa/pip/issues/468 / https://github.com/pypa/pip/pull/735 https://github.com/erikrose/peep/issues/7
Summary: Consider using a fork of the Heroku Python buildpack that uses peep rather than pip → Use peep on Heroku instead of pip
Fingers crossed https://github.com/pypa/pip/pull/3137 should be in pip v8, which once the python buildpack is updated means we can just use that instead :-)
Pip v8 (due to be released in ~January) will have native support for hashes.
Summary: Use peep on Heroku instead of pip → Use pip v8's hash support on Heroku
I have WIP branches against Treeherder, heroku-buildpack-python and virtualenv for supporting pip v8, to expedite this. Once v8 is released I'll open some PRs.
Pip v8 was released today. virtualenv has been updated to include it in the new 14.0.0 release. I've opened a PR against the Heroku Python buildpack, to update there too: https://github.com/heroku/heroku-buildpack-python/pull/270
Summary: Use pip v8's hash support on Heroku → Use pip v8's hash support instead of peep
Created attachment 8711023 [details] [review] [treeherder] mozilla:pip-require-hashes > mozilla:master
Attachment #8711023 - Flags: review?(mdoglio) → review+
Commits pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/a83f93029644ca370ef44c5d708ac02e7cd687d6 Bug 1213230 - Use pip 8's require-hashes feature instead of peep As of pip 8, peep has now been integrated into pip. Migrating from peep to this native feature has several advantages: * It avoids the complexity/learning curve of using a wrapper around pip. * It means we do not need to fork the official Heroku Python buildpack (which handles pip installation of requirements files) in order to use hash verification on Heroku. (Once the buildpack updates to pip 8.) * Omitted sub-dependencies result in install-time errors rather than the user discovering omissions at run-time. * pip's native caching is used, and all packages are installed in one pip invocation, so it's significantly faster. * It has better handling of errors and corner cases. Key facts about the native feature: * hash-checking mode is enabled if at least one hash is found in the requirements files passed to pip, or can be force enabled by passing `--requires-hashes` when running `pip install`. * Once enabled, hash-checking mode enforces that all packages: - are pinned to a specific version - have hashes listed - have all sub-dependencies specified * Older versions of pip will error out if either `--require-hashes` or the requirements file `--hash` syntax is used, meaning it's not possible to accidentally lose hash-checking protection if the pip used is older than expected. For more details, see: https://pip.pypa.io/en/stable/user_guide/#hash-checking-mode https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode The pip version on Travis and in the Vagrant virtualenv has been updated to 8.0.2 in bug 1241144, and the stage/prod virtualenv in bug 1241519. The Heroku Python buildpack pip was updated in bug 1241909. The requirements files hashes were ported using `peep port`, and then comments/URLs re-added by hand. https://github.com/mozilla/treeherder/commit/b2ecb99390185ed7883b2334c1b11855ddd0811b Bug 1213230 - Remove peep.py Since it's now unused.
I've purged the virtualenvs and pre-updated the update.py scripts (to avoid the need to double deploy) for both stage and prod: [email@example.com ~]$ ths [firstname.lastname@example.org treeherder-service]$ sudo git fetch --quiet origin master && sudo git reset --hard FETCH_HEAD HEAD is now at b2ecb99 Bug 1213230 - Remove peep.py [email@example.com treeherder-service]$ cd .. [firstname.lastname@example.org treeherder.allizom.org]$ ll total 12 -rw-r--r-- 1 treeherder treeherder 1290 Dec 29 15:00 treeherder-env.sh drwxr-xr-x 14 treeherder treeherder 4096 Feb 11 18:34 treeherder-service drwxr-xr-x 6 root root 4096 Feb 11 18:33 venv [email@example.com treeherder.allizom.org]$ sudo rm -r venv/ [firstname.lastname@example.org treeherder-service]$ thp [email@example.com treeherder-service]$ sudo git fetch --quiet origin master && sudo git reset --hard FETCH_HEAD HEAD is now at b2ecb99 Bug 1213230 - Remove peep.py [firstname.lastname@example.org treeherder-service]$ cd .. [email@example.com treeherder.mozilla.org]$ ll total 12 -rw-r--r-- 1 treeherder treeherder 1243 Feb 4 14:19 treeherder-env.sh drwxr-xr-x 14 treeherder treeherder 4096 Feb 3 22:54 treeherder-service drwxr-xr-x 6 root root 4096 Feb 3 22:54 venv [firstname.lastname@example.org treeherder.mozilla.org]$ sudo rm -r venv/ [email@example.com treeherder-service]$ And deployed to stage: http://treeherderadm.private.scl3.mozilla.com/chief/treeherder.stage/logs/stage.1455540640
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.