Change from using vendor packages to peep

RESOLVED FIXED

Status

support.mozilla.org
Code Quality
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: deanj, Assigned: deanj)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: p=2 u=dev s= c=deploys)

(Assignee)

Description

4 years ago
We should switch from using our vendor package system to using peep. The reason we originally used the vendor system was that some dependencies could change, even within their versions, and we needed those dependencies completely frozen. Peep solves this by letting you submit a commit hash that it uses for installation, so that the packages are always consistent.

We will have a new requirements file that keeps the hashes for all of our (current) vendor packages and will be installed with: `peep install -r requirements/peep.txt`. The TravisCI script will also have to be updated to match our new deployment strategy.

Reference:
* Peep: https://github.com/erikrose/peep

Other projects using peep:
* https://github.com/mozilla/dxr
* https://github.com/mozilla/socorro
(Assignee)

Comment 1

4 years ago
I'm going to be working on this bug, as it will greatly improve load times in my vagrant environment. I'm setting this at a value of 2 points.
Assignee: nobody → djohnson
(Assignee)

Updated

4 years ago
Whiteboard: p=2 u=dev
Whiteboard: p=2 u=dev → p=2 u=dev s= c=deploys
(Assignee)

Comment 2

4 years ago
Currently, I've got a (peep) requirements file that references all packages from the vendor/src directory. The main problem I'm running into now, is how to generate the requirements for the vendor/packages directory. This is the approach I've tried so far, which hasn't yielded *identical* packages to the ones we currently have:

1. Look up version number if I can find it in package.
2. Attempt to download via pypi.

I say *identical* because the packages seemed to be nearly the same, but I was getting different md5 hashes for the vendor and downloaded packages.

Any thoughts on how I might be able to get the exact packages, or do you think it might be better to find as similar as I can and thoroughly test the codebase once they are being used?
Flags: needinfo?(rrosario)
Flags: needinfo?(mcooper)
Pick a package and check to see if all the files in vendor are the same as in the package. I bet there are some additional pip-created metadata files or .pyc files or something like that which is throwing off your md5 checksum.
Clearing needinfo on me. I have nothing new to add here.
Flags: needinfo?(mcooper)
I've got this half-done for Fjord now. Using a virtual environment rather than vendor makes manage.py in the vagrant development environment work a *lot* faster.

I still have a bunch of things to do, but I'm working through them in phases.

All that work is being done in bug #987801.
(Assignee)

Updated

4 years ago
Status: NEW → ASSIGNED
(Assignee)

Updated

4 years ago
Blocks: 905834
(Assignee)

Comment 7

4 years ago
Now on stage. This needs to be heavily tested.
How should this be tested? Is there a test plan?
(Assignee)

Updated

4 years ago
Depends on: 1059996
(Assignee)

Updated

4 years ago
Flags: needinfo?(rrosario)
(Assignee)

Comment 9

4 years ago
This is going to sit on stage over the weekend. If all goes well, we should merge this into master next week.
(Assignee)

Updated

4 years ago
Depends on: 1065152
Depends on: 1067586
Blocks: 1068237
(Assignee)

Updated

4 years ago
Depends on: 1069516
(Assignee)

Comment 11

4 years ago
Landed on master, and master was pushed to prod. All looks good!

https://github.com/mozilla/kitsune/pull/2083
(Assignee)

Updated

4 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Depends on: 1070020
You need to log in before you can comment on or make changes to this bug.