Closed Bug 629029 Opened 9 years ago Closed 9 years ago

get daily correlation reports cron into Socorro SVN

Categories

(Socorro :: General, task)

x86_64
Linux
task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: rhelmer, Assigned: jabba)

References

Details

Attachments

(1 file, 2 obsolete files)

Most cron jobs are now part of the normal Socorro release (bug 608190) but there are two which have not been refactored and brought into google code SVN.

Together, these generate the files that go into http://people.mozilla.com/crash_analysis/20110125/

1) cron_daily_reports.sh
2) cron_libraries.sh

The first uses http://code.google.com/p/socorro/source/browse/tags/releases/1.7.6_r2892_20110121/socorro/cron/dailyUrl.py to generate e.g. http://people.mozilla.com/crash_analysis/20110125/20110125-pub-crashdata.csv.gz

The second is a bunch of standalone bash and psql, and generates all the .txt and .txt.gz files in http://people.mozilla.com/crash_analysis/20110125/ such as 20110125_Firefox_3.5.16-core-counts.txt

We may want to just combine these into one script, at the very least I think we want to use Python for #2 so it can take advantage of configuration (products and versions are hardcoded into the script right now).
Assignee: nobody → rhelmer
Blocks: 574711
1.  Bring it in ASAP
2.  Improve and refactor (probably a separate bug)
Blocks: 524851
Target Milestone: --- → 1.7.7
This brings cron_daily_reports.sh into SVN. It does not really change anything, just removes hardcoded values and shares more code with our other cron jobs.
Attachment #507175 - Flags: review?(laura)
Attachment #507175 - Flags: feedback?(jdow)
Attached patch bring cron_libraries into SVN (obsolete) — Splinter Review
This is a (slightly) cleaned-up version of cron_libraries.sh

Note that this depends on scripts in /data/crash-data-tools which now live in private SVN. Might be nice to liberate these if they don't need to be private (could possibly live alongside the Hadoop versions in SVN analysis/src/python/)

I'd like to test this and add more error checking, but will leave major refactoring for a separate bug (see comment 1).
Attachment #507180 - Flags: review?(laura)
Attachment #507180 - Flags: feedback?(jdow)
So if I understand it correctly, all the hardcoded values for this will end up as variables in /etc/common.conf ? That sounds like a good idea. Is there a friendly way we can test this script without clobbering data anywhere?
If it sources /etc/common.conf, I believe that file has the database password in it. Then you could add the -P to the command that connects to postgres, and eliminate my dependency of scattering .pgpass files around.
(In reply to comment #4)
> So if I understand it correctly, all the hardcoded values for this will end up
> as variables in /etc/common.conf ? That sounds like a good idea. Is there a
> friendly way we can test this script without clobbering data anywhere?

Right. Yes, we can test on staging, and easily configure these to point wherever we want ;)
(In reply to comment #5)
> If it sources /etc/common.conf, I believe that file has the database password
> in it. Then you could add the -P to the command that connects to postgres, and
> eliminate my dependency of scattering .pgpass files around.

OK will do this in the next patch (I would like to add more error-checking anyway)
Attachment #507175 - Flags: review?(laura) → review+
Attachment #507180 - Flags: review?(laura) → review+
(In reply to comment #7)
> (In reply to comment #5)
> > If it sources /etc/common.conf, I believe that file has the database password
> > in it. Then you could add the -P to the command that connects to postgres, and
> > eliminate my dependency of scattering .pgpass files around.
> 
> OK will do this in the next patch (I would like to add more error-checking
> anyway)

Hmm so two things:

1) -P doesn't do that for psql:
  -P, --pset=VAR[=ARG]     set printing option VAR to ARG (see \pset command)

2) that will make the password visible in the process table (e.g. "ps awux")

Looks like this will do it:

export PGPASSWORD=$databasePassword

We could have socorrorc set this for us I suppose, but will hold off on that for now and just do it in this script.
Committed r2902

Going over these again, we don't really need to do anything special to catch the errors as far as I can tell. We should get email if anything fails as-is.

We should not redirect the output of these scripts, if they are too chatty we can make them less so.

jabba can you please add these to the staging crontab? We'll also need to set these in common.conf (I was thinking of perhaps also having a cron.conf, let me know what you think about that):

PUB_USER
PUB_SERVER
PUB_LOCATION

PRIV_USER
PRIV_SERVER
PRIV_LOCATION

You should be able to get all of these values from puppet SVN in files/generic/prod/data-bin/cron_daily_reports.sh
Attachment #507175 - Attachment is obsolete: true
Attachment #507180 - Attachment is obsolete: true
Attachment #507175 - Flags: feedback?(jdow)
Attachment #507180 - Flags: feedback?(jdow)
Assignee: rhelmer → jdow
So if I understand correctly, I just need to add these to the staging crontab and set the values on common.conf. The problem is, I don't think I have values for those keys right now. I'd need a few servers to upload those to, right? although I guess we could just scp the files to localhost for staging, right?
(In reply to comment #10)
> So if I understand correctly, I just need to add these to the staging crontab
> and set the values on common.conf. The problem is, I don't think I have values
> for those keys right now. I'd need a few servers to upload those to, right?
> although I guess we could just scp the files to localhost for staging, right?

Yes I think using localhost to test would be just fine.
Target Milestone: 1.7.7 → 1.7.8
Depends on: 641176
Added:
05 00 * * * socorro /data/socorro/application/scripts/crons/cron_libraries.sh
55 03 * * * socorro /data/socorro/application/scripts/crons/cron_daily_reports.sh

to the crontab and:

export PUB_USER='socorro'
export PUB_SERVER='127.0.0.1'
export PUB_LOCATION='/home/socorro/crash_analysis/'
export PRIV_USER='socorro'
export PRIV_SERVER='127.0.0.1'
export PRIV_LOCATION='/home/socorro/crash_urls/'

to common.conf on staging. This should just work and the files will end up in socorro's home directory. Once we've verified this, I can do the same on prod during the next release window.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Blocks: 640238
Component: Socorro → General
Product: Webtools → Socorro
You need to log in before you can comment on or make changes to this bug.