894493 - use peep for installing dependencies

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Lonnen :lonnen]

ErikRose's peep adds an extra layer of assurance at little cost. It's backwards compatible for devs installing from pip, but our CI can use it to verify the sha of the packages we're pulling down.

https://pypi.python.org/pypi/peep/

Comment 1

[Robert Helmer [:rhelmer]]

Spent a little time on this:
https://github.com/rhelmer/socorro/compare/bug894493-use-peep-for-deps

Since we can't trust peep to verify itself, I've simply checked in the latest version to ./tools/ and use pip to install it. From there, we use peep and SHA 256 hashes to verify packages from pypi.

Two problems right now:
 
1) peep throws an exception when it tries to print info for sqlalchemy-citext (non-standard filename, contains more than one "-")
2) configman needs to come from pypi not git

We could work around #1 (only happens if there's no hash set up yet, want it fixed since it makes upgrading a pain but could ship with it I think).

We don't have a good workaround for #2 - :peterbe can we get the right version of configman up on pypi?

Comment 2

[Erik Rose [:erik][:erikrose]]

#1 is fixed in peep 0.4.

Comment 3

[Robert Helmer [:rhelmer]]

r? anybody - https://github.com/mozilla/socorro/pull/1457

Note that we check peep in and install it from there because we the whole point here is to not trust pypi, and we can't trust a version of peep that we download from there to verify itself ;)

We *could* download peep from pypi and use a different tool, but I don't really see the point since peep is just a tiny wrapper around pypi, anyway.

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #3)
> r? anybody - https://github.com/mozilla/socorro/pull/1457
> 
> Note that we check peep in and install it from there because we the whole
> point here is to not trust pypi, and we can't trust a version of peep that
> we download from there to verify itself ;)
> 
> We *could* download peep from pypi and use a different tool, but I don't
> really see the point since peep is just a tiny wrapper around pypi, anyway.

Er, "peep is just a tiny wrapper around pip, anyway" imts

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/8ba14b7776dabcad0ade1bbbf6e7f55c0722b561
bug 894493 - use peep for dependencies

https://github.com/mozilla/socorro/commit/2da829f054b85184604ea99b97dac0af34c21e31
Merge pull request #1457 from rhelmer/bug894493-use-peep-for-deps

[DO NOT MERGE] bug 894493 - use peep for dependencies

Comment 6

[Robert Helmer [:rhelmer]]

Although this worked fine on leeroy, getting this test failure on socorro-release:

FAIL: test_get_count_by_day (socorro.unittest.external.postgresql.test_crashes.IntegrationTestCrashes)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/jenkins/jobs/socorro-release/workspace/socorro/unittest/external/postgresql/test_crashes.py", line 634, in test_get_count_by_day
    self.assertEquals(res, expected)
AssertionError: {'hits': {'2013-08-29': 2L}, 'total': 1} != {'hits': {'2013-08-29': 3}, 'total': 1}

Comment 7

[Robert Helmer [:rhelmer]]

Going to try wiping the workspace before backing out

Comment 8

[Robert Helmer [:rhelmer]]

OK leeroy's failing on the backout patch! I don't even:

https://ci.mozilla.org/job/socorro-github/1514/console

Comment 9

[[DEACTIVATED] Adrian Gaudebert]

I can't reproduce locally. CC'ing Shuhao as he wrote this unit test.

Comment 10

[Shuhao]

Found the issue. Is on https://github.com/mozilla/socorro/blob/master/socorro/unittest/external/postgresql/test_crashes.py#L615

That now is sometimes one day behind UTC now, which results in a 2 instead of 3. Fix coming soon.

Comment 11

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/03c688b767c3a25f4c300b340b6c70b1c055c9ef
fixes bug 894493 - build requirements against python 2.6

https://github.com/mozilla/socorro/commit/e3a78d1b9d4fb1ef2662e0e38db74f6c4d7c66eb
Merge pull request #1460 from rhelmer/bug894493-2.6-deps

fixes bug 894493 - build requirements against python 2.6

Comment 12

[Erik Rose [:erik][:erikrose]]

Hooray! :-D