602559 - build script for Socorro

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Robert Helmer [:rhelmer]]

I've set Hudson up to run a series of shell commands, to:

1) run PHPUnit tests
2) set up virtualenv and install deps
3) run unittests, code coverage, pylint, etc.

In the near future it can also do things like:
4) build packages
5) install to staging environemnt
6) run automated QA (e.g. Selenium), acceptance tests, etc.

We need to get this out of Hudson and into a shell script, which lives with the Socorro sources. This is important because Hudson doesn't have version control for changes, so we want to keep it's job as simple as possible. 

Also, it simplifies setup for developers, and makes sure we're doing things the same way staging and (eventually) production do.

Comment 1

[Robert Helmer [:rhelmer]]

Attachment: scripts/build.sh for socorro

This is exactly what https://hudson.mozilla.org/job/socorro/ is currently doing, the first 3 steps of comment 0.

I looked at how other projects use Hudson, AMO uses "scripts/build.sh" and Sync uses a Makefile (with comments about switching to zc.buildout at some point). The way code coverage is done here is lifted from Sync. 

Let's keep it simple and just drop in a scripts/build.sh, if we outgrow this we can look at going directly to something modern like buildout. 

We should make some other improvements like using a local pypi mirror, especially if we depend on this for building staging/production packages. Not super critical at this moment though.

Comment 2

[Robert Helmer [:rhelmer]]

Spoke to lars on irc, checking in:

Committed revision 2604.

Hudson's only build step is now:

trunk/scripts/build.sh

Comment 3

[Robert Helmer [:rhelmer]]

Oops, had to set executable:

Committed revision 2605.

Now it works:

https://hudson.mozilla.org/job/socorro/70/

Anything we want Hudson to do should now go into this script.

Comment 4

[K Lars Lohn [:lars] [:klohn]]

I notice in your script that you use easy_install to install 'thrift'.  Right now, because of some complications, we use a local copy of thrift installed under 'thirdparty' of our project root.  This was a compromise because of some difficulty that occurred way back when I was on medical leave.  I do not have the details, I believe ozten set it up.  We should reconcile this oddity in our tree to make sure we're using the proper version of thrift.

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to comment #4)
> I notice in your script that you use easy_install to install 'thrift'.  Right
> now, because of some complications, we use a local copy of thrift installed
> under 'thirdparty' of our project root.  This was a compromise because of some
> difficulty that occurred way back when I was on medical leave.  I do not have
> the details, I believe ozten set it up.  We should reconcile this oddity in our
> tree to make sure we're using the proper version of thrift.

Ah thanks that makes sense; I believe that the local one is being used due to the way the PYTHONPATH is being hardcoded in the script.

I'll compare the pypi version against our local version, if we just need to carry around a local one and don't need to easy_install a new one that's fine. I just want this script to accurately reflect where we are now.

Comment 6

[Austin King [:ozten]]

(In reply to comment #4)
I wanted to deploy using virtualenvs, but IT pushed back. We can and should revisit these decisions.

Comment 7

[Robert Helmer [:rhelmer]]

(In reply to comment #6)
> (In reply to comment #4)
> I wanted to deploy using virtualenvs, but IT pushed back. We can and should
> revisit these decisions.

I agree that virtualenv probably makes more sense than using RPMs for these types of dependencies, but we're still pretty early in the game. See bug 605185 comment 0 for more thoughts on where we need to get to first.

My main concern right now is that we're doing the same thing in staging and production. Doing the same thing in dev could potentially save us a lot of time (e.g. a unit test works in newer virtualenv-installed deps but not the old ones from the RPM in staging/prod), but it's so much less painful to use virtualenv from Hudson I think it's worth it for now.

Comment 8

[Robert Helmer [:rhelmer]]

Hudson was temporarily broken since we moved 1.7reversion branch to trunk, I've copied build.sh from 1.8abandoned branch and now Hudson is going again:

Committed revision 2666.