751654 - ship python requirements with socorro

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Robert Helmer [:rhelmer]]

lonnen and I were just discussing on IRC how painful it is to maintain our Python libraries as RPM, especially with a distro like RHEL this means we need to spend a lot of time building RPMs from pypi and giving them to IT, and blocking releases on these type of out-of-band activities. It's painful enough that we don't really keep things up to date.

Instead I propose we have the Makefile do:

1) pip install -r requirements.txt -> socorro-virtualenv
2) copy socorro-virtualenv/(...)/site-packages/ -> /data/socorro/thirdparty/

This means that things will Just Work when you add a dependency to requirements.txt, in all environments (local, vagrant, jenkins, dev/stage/prod).

Jenkins will build the socorro.tar.gz with these baked on, so IT doesn't need to worry about running pip and having it pollute the system Python dir (and having to track that etc)

Comment 1

[Robert Helmer [:rhelmer]]

r? https://github.com/mozilla/socorro/pull/553

Comment 2

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/61810a2714a5257d43171a9bfbe59eafc43d6ca1
bug 751654 - ship site-packages from virtualenv in thirdparty dir

https://github.com/mozilla/socorro/commit/970eae86bd52d4e36bcc17ce44db5553bbda5209
Merge pull request #553 from rhelmer/bug751654-ship-requirements

bug 751654 - ship site-packages from virtualenv in thirdparty dir

Comment 3

[Robert Helmer [:rhelmer]]

r? https://github.com/mozilla/socorro/pull/562

After discussing w/ peterbe:

* use requirements instead of submodule for configman
* pin versions in requirements.txt
* remove everything in thirdparty/ use requirements only

Comment 4

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/a2933c481f413df8860a138a854d8b8dff22f5a9
bug 751654 - pin versions and manage thirdparty with requirements.txt only

https://github.com/mozilla/socorro/commit/e84658dcc2881b9ad23bda67ed519953640f1686
Merge pull request #562 from rhelmer/bug751654-cleanup-reqs

bug 751654 - pin versions and manage thirdparty with requirements.txt on...

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/5c55a4900b02e72e6807ec09000445d4af656340
bug 751654 - make psycopg2 work for RHEL6 with postgres 9.0 RPM

https://github.com/mozilla/socorro/commit/b3b6e0f6f11d66ab7bd377f69455111a4639a264
Merge pull request #565 from rhelmer/master

bug 751654 - make psycopg2 work for RHEL6 with postgres 9.0 RPM

Comment 6

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/e7ea2a41c2e605ca3c3f7e582a46d73390f25e56
bug 751654 - slight change in hbase lib

https://github.com/mozilla/socorro/commit/c4f0e17461494887cdcef9884a2f790cda797258
Merge pull request #566 from rhelmer/master

bug 751654 - slight change in hbase lib

Comment 7

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/a49a4bbd787210002b158f4617067fd3d31ccf4f
bug 751654 - slight difference in latest psycopg2

Comment 8

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/380c00efa754f47167d89b21c59b3902ad90bde7
bug 751654 - setupdb needs thirdparty on PYTHONPATH now that we do not expect the system to provide it

https://github.com/mozilla/socorro/commit/e3961922ae147284c62f50a7e22776855c64925e
Merge pull request #568 from rhelmer/master

bug 751654 - setupdb needs thirdparty on PYTHONPATH

Comment 9

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/eb82512369d38cefaac8c523abb97b84403d8f38
bug 751654 - ensure virtualenv exists

https://github.com/mozilla/socorro/commit/74f9c1b3523d8e25c9d1fb42c262b79af793f34c
Merge pull request #569 from rhelmer/master

bug 751654 - ensure virtualenv exists

Comment 10

[Robert Helmer [:rhelmer]]

This is mostly qa-, although we'll want to double-check that all the usual things work (generate a crash and send it in, make sure a report is available for it shortly thereafter)

Comment 11

[Matt Brandt [:mbrandt]]

Thanks rhelmer - I'll bump this to [qa-] then. Our automaton and integration tests should provide the appropriate coverage.