Closed Bug 987801 Opened 8 years ago Closed 8 years ago

implement peep-installed requirements.txt file for dependencies

Categories

(Input Graveyard :: Code Quality, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: willkg, Assigned: willkg)

References

Details

(Whiteboard: u=dev c=codequality p=5 s=input.2014q3)

peep <https://github.com/erikrose/peep> works like pip, but lets you specify checksums for the individual requirements giving you the confidence that future installations of those dependencies are the same as the one you made the checksum for.

Switching to using peep and a requirements file would alleviate many issues we currently have with maintaining vendor/:

1. easier on contributors because they don't have to download entire repositories (all the git submodules) which takes a really long time on low-bandwidth connections (Joshua-S just hit this)

2. reduces the speed at which our repository grows (we have megabytes of data in vendor/packages/ and a lot of flux as we update dependencies there)

3. reduces developer time spent dealing with manipulating git submodules (I've spent my fair share of time trying to get things ungoofed after goofing them)

4. makes it much easier to compare what we're using now with the latest available which makes it much easier to keep up-to-date


This bug covers looking into whether it's feasible to switch to peep (pretty sure it is), figuring out how to migrate from where we are now with vendor/ to peep requirements files and figuring out how to change our deployment scripts.
Putting this in 2014q2. If I don't get to it in 2014q2, I'll make it a P1 for 2014q3 because this is really important and fixes a bunch of ongoing maintenance issues.
Component: Backend → Code Quality
Priority: -- → P3
Whiteboard: u=dev c=codequality p= s=input.2014q2
Bumping this to 2014q3. It requires this to be fixed:

https://github.com/erikrose/peep/issues/19
Whiteboard: u=dev c=codequality p= s=input.2014q2 → u=dev c=codequality p= s=input.2014q3
That fix to peep landed. Pretty sure we can move forward with this.
Dean is working on peepify which might simplify a lot of the work required here:

https://github.com/dean/peepify

Definitely worth trying to use that and improving it if we can.
Grabbing this one as I'm working on it now.

1. I've fixed some issues in peepify that make it work a bit better.
2. I had to update django-mozilla-product-details to a version that doesn't import product_details in setup.py.

That probably covers git submodules in vendor/src/

Outstanding things:

1. vendor/packages/
2. fix manage.py to either add a venv/ directory to sys.path OR do the site dir thing with vendor/
3. see if updating from a tarball in a virtual environment with peep handles files that went away
Assignee: nobody → willkg
PR: https://github.com/mozilla/fjord/pull/326

Still outstanding:

3. see if updating from a tarball in a virtual environment with peep handles files that went away

Also:

4. figure out how all this works with deployment to servers
Also:

5. Go through requirements.txt and nix as many "install-from-github-tarball" items as we can because that way is painful. We should minimize those as much as possible.
For handling the problem of mismatch between the versions of packages installed in a virtualenv environment and the requirements.txt either to due to updates to the repository or switching branches, we could use manage.py for the notifying the mismatch as an error. This would make a lot of sense because in a dev environment manage.py is the most run command and looks like a very good way to notify the developer to run something like "pip install --reinstall -r /path/to/requirements.txt"
PR 326 landed in master in:

https://github.com/mozilla/fjord/commit/58579abb
https://github.com/mozilla/fjord/commit/248b0abc
https://github.com/mozilla/fjord/commit/5db57bc3

This incorporates L Guruprasad's idea in comment #8.

Changing this from a research bug to an implementation bug. Given that this is pretty intensive, I'm just going to make it a 5-point bug.


Still outstanding:

3. See if updating from a tarball in a virtual environment with peep handles files that went away.

4. Figure out how all this works with deployment to servers.

5. Go through requirements.txt and nix as many "install-from-github-tarball" items as we can because that way is painful. We should minimize those as much as possible.
Status: NEW → ASSIGNED
Summary: [research] switch from vendor/ to peep-installed requirements.txt file → switch from vendor/ to peep-installed requirements.txt file
Whiteboard: u=dev c=codequality p= s=input.2014q3 → u=dev c=codequality p=5 s=input.2014q3
Blocks: 1048459
No longer depends on: 1048459
Spinning the outstanding parts into separate bugs and tweaking the summary to reduce the scope of this bug a little.

Item 3: bug #1048459

Item 4: bug #1048461

Item 5: bug #1048462

Closing this out as FIXED.
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Summary: switch from vendor/ to peep-installed requirements.txt file → implement peep-installed requirements.txt file for dependencies
Product: Input → Input Graveyard
You need to log in before you can comment on or make changes to this bug.