629029 - get daily correlation reports cron into Socorro SVN

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Robert Helmer [:rhelmer]]

Most cron jobs are now part of the normal Socorro release (bug 608190) but there are two which have not been refactored and brought into google code SVN.

Together, these generate the files that go into http://people.mozilla.com/crash_analysis/20110125/

1) cron_daily_reports.sh
2) cron_libraries.sh

The first uses http://code.google.com/p/socorro/source/browse/tags/releases/1.7.6_r2892_20110121/socorro/cron/dailyUrl.py to generate e.g. http://people.mozilla.com/crash_analysis/20110125/20110125-pub-crashdata.csv.gz

The second is a bunch of standalone bash and psql, and generates all the .txt and .txt.gz files in http://people.mozilla.com/crash_analysis/20110125/ such as 20110125_Firefox_3.5.16-core-counts.txt

We may want to just combine these into one script, at the very least I think we want to use Python for #2 so it can take advantage of configuration (products and versions are hardcoded into the script right now).

Comment 1

[Laura Thomson :laura]

1.  Bring it in ASAP
2.  Improve and refactor (probably a separate bug)

Comment 2

[Robert Helmer [:rhelmer]]

Attachment: bring cron_daily_reports into SVN

This brings cron_daily_reports.sh into SVN. It does not really change anything, just removes hardcoded values and shares more code with our other cron jobs.

Comment 3

[Robert Helmer [:rhelmer]]

Attachment: bring cron_libraries into SVN

This is a (slightly) cleaned-up version of cron_libraries.sh

Note that this depends on scripts in /data/crash-data-tools which now live in private SVN. Might be nice to liberate these if they don't need to be private (could possibly live alongside the Hadoop versions in SVN analysis/src/python/)

I'd like to test this and add more error checking, but will leave major refactoring for a separate bug (see comment 1).

Comment 4

[Justin Dow [:jabba]]

So if I understand it correctly, all the hardcoded values for this will end up as variables in /etc/common.conf ? That sounds like a good idea. Is there a friendly way we can test this script without clobbering data anywhere?

Comment 5

[Justin Dow [:jabba]]

If it sources /etc/common.conf, I believe that file has the database password in it. Then you could add the -P to the command that connects to postgres, and eliminate my dependency of scattering .pgpass files around.

Comment 6

[Robert Helmer [:rhelmer]]

(In reply to comment #4)
> So if I understand it correctly, all the hardcoded values for this will end up
> as variables in /etc/common.conf ? That sounds like a good idea. Is there a
> friendly way we can test this script without clobbering data anywhere?

Right. Yes, we can test on staging, and easily configure these to point wherever we want ;)

Comment 7

[Robert Helmer [:rhelmer]]

(In reply to comment #5)
> If it sources /etc/common.conf, I believe that file has the database password
> in it. Then you could add the -P to the command that connects to postgres, and
> eliminate my dependency of scattering .pgpass files around.

OK will do this in the next patch (I would like to add more error-checking anyway)

Comment 8

[Robert Helmer [:rhelmer]]

(In reply to comment #7)
> (In reply to comment #5)
> > If it sources /etc/common.conf, I believe that file has the database password
> > in it. Then you could add the -P to the command that connects to postgres, and
> > eliminate my dependency of scattering .pgpass files around.
> 
> OK will do this in the next patch (I would like to add more error-checking
> anyway)

Hmm so two things:

1) -P doesn't do that for psql:
  -P, --pset=VAR[=ARG]     set printing option VAR to ARG (see \pset command)

2) that will make the password visible in the process table (e.g. "ps awux")

Looks like this will do it:

export PGPASSWORD=$databasePassword

We could have socorrorc set this for us I suppose, but will hold off on that for now and just do it in this script.

Comment 9

[Robert Helmer [:rhelmer]]

Attachment: [checked in] bring correlation cron jobs into SVN

Committed r2902

Going over these again, we don't really need to do anything special to catch the errors as far as I can tell. We should get email if anything fails as-is.

We should not redirect the output of these scripts, if they are too chatty we can make them less so.

jabba can you please add these to the staging crontab? We'll also need to set these in common.conf (I was thinking of perhaps also having a cron.conf, let me know what you think about that):

PUB_USER
PUB_SERVER
PUB_LOCATION

PRIV_USER
PRIV_SERVER
PRIV_LOCATION

You should be able to get all of these values from puppet SVN in files/generic/prod/data-bin/cron_daily_reports.sh

Comment 10

[Justin Dow [:jabba]]

So if I understand correctly, I just need to add these to the staging crontab and set the values on common.conf. The problem is, I don't think I have values for those keys right now. I'd need a few servers to upload those to, right? although I guess we could just scp the files to localhost for staging, right?

Comment 11

[Robert Helmer [:rhelmer]]

(In reply to comment #10)
> So if I understand correctly, I just need to add these to the staging crontab
> and set the values on common.conf. The problem is, I don't think I have values
> for those keys right now. I'd need a few servers to upload those to, right?
> although I guess we could just scp the files to localhost for staging, right?

Yes I think using localhost to test would be just fine.

Comment 12

[Justin Dow [:jabba]]

Added:
05 00 * * * socorro /data/socorro/application/scripts/crons/cron_libraries.sh
55 03 * * * socorro /data/socorro/application/scripts/crons/cron_daily_reports.sh

to the crontab and:

export PUB_USER='socorro'
export PUB_SERVER='127.0.0.1'
export PUB_LOCATION='/home/socorro/crash_analysis/'
export PRIV_USER='socorro'
export PRIV_SERVER='127.0.0.1'
export PRIV_LOCATION='/home/socorro/crash_urls/'

to common.conf on staging. This should just work and the files will end up in socorro's home directory. Once we've verified this, I can do the same on prod during the next release window.