987801 - implement peep-installed requirements.txt file for dependencies

Status: RESOLVED FIXED

(Input Graveyard :: Code Quality, defect, P3)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

peep <https://github.com/erikrose/peep> works like pip, but lets you specify checksums for the individual requirements giving you the confidence that future installations of those dependencies are the same as the one you made the checksum for.

Switching to using peep and a requirements file would alleviate many issues we currently have with maintaining vendor/:

1. easier on contributors because they don't have to download entire repositories (all the git submodules) which takes a really long time on low-bandwidth connections (Joshua-S just hit this)

2. reduces the speed at which our repository grows (we have megabytes of data in vendor/packages/ and a lot of flux as we update dependencies there)

3. reduces developer time spent dealing with manipulating git submodules (I've spent my fair share of time trying to get things ungoofed after goofing them)

4. makes it much easier to compare what we're using now with the latest available which makes it much easier to keep up-to-date


This bug covers looking into whether it's feasible to switch to peep (pretty sure it is), figuring out how to migrate from where we are now with vendor/ to peep requirements files and figuring out how to change our deployment scripts.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Putting this in 2014q2. If I don't get to it in 2014q2, I'll make it a P1 for 2014q3 because this is really important and fixes a bunch of ongoing maintenance issues.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Bumping this to 2014q3. It requires this to be fixed:

https://github.com/erikrose/peep/issues/19

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

That fix to peep landed. Pretty sure we can move forward with this.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Dean is working on peepify which might simplify a lot of the work required here:

https://github.com/dean/peepify

Definitely worth trying to use that and improving it if we can.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this one as I'm working on it now.

1. I've fixed some issues in peepify that make it work a bit better.
2. I had to update django-mozilla-product-details to a version that doesn't import product_details in setup.py.

That probably covers git submodules in vendor/src/

Outstanding things:

1. vendor/packages/
2. fix manage.py to either add a venv/ directory to sys.path OR do the site dir thing with vendor/
3. see if updating from a tarball in a virtual environment with peep handles files that went away

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

PR: https://github.com/mozilla/fjord/pull/326

Still outstanding:

3. see if updating from a tarball in a virtual environment with peep handles files that went away

Also:

4. figure out how all this works with deployment to servers

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

Also:

5. Go through requirements.txt and nix as many "install-from-github-tarball" items as we can because that way is painful. We should minimize those as much as possible.

Comment 8

[L. Guruprasad]

For handling the problem of mismatch between the versions of packages installed in a virtualenv environment and the requirements.txt either to due to updates to the repository or switching branches, we could use manage.py for the notifying the mismatch as an error. This would make a lot of sense because in a dev environment manage.py is the most run command and looks like a very good way to notify the developer to run something like "pip install --reinstall -r /path/to/requirements.txt"

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

PR 326 landed in master in:

https://github.com/mozilla/fjord/commit/58579abb
https://github.com/mozilla/fjord/commit/248b0abc
https://github.com/mozilla/fjord/commit/5db57bc3

This incorporates L Guruprasad's idea in comment #8.

Changing this from a research bug to an implementation bug. Given that this is pretty intensive, I'm just going to make it a 5-point bug.


Still outstanding:

3. See if updating from a tarball in a virtual environment with peep handles files that went away.

4. Figure out how all this works with deployment to servers.

5. Go through requirements.txt and nix as many "install-from-github-tarball" items as we can because that way is painful. We should minimize those as much as possible.

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

Spinning the outstanding parts into separate bugs and tweaking the summary to reduce the scope of this bug a little.

Item 3: bug #1048459

Item 4: bug #1048461

Item 5: bug #1048462

Closing this out as FIXED.