821930 - [socorro stage] enable crontabber on staging socorroadm

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Robert Helmer [:rhelmer]]

We have a new crontab manager, which should handle failure and retrying better.

I'll post a new crontab momentarily.

Comment 1

[Robert Helmer [:rhelmer]]

New staging crontab could be as simple as this (and even shorter as we move more stuff into crontabber):

MAILTO="[e-mail]"

# Socorro cron job manager
*/5 * * * * socorro /data/socorro/application/scripts/crons/crontabber.sh

# Generate data for /status page
*/5 * * * * socorro /data/socorro/application/scripts/crons/cron_status.sh

# Modulelist report for crash-analysis
00 17 * * * socorro export JAVA_HOME=/usr/lib/jvm/java-1.6.0;/data/socorro/application/scripts/crons/cron_modulelist.sh

# Correlation report for crash-analysis
05 00 * * * socorro /data/socorro/application/scripts/crons/cron_libraries.sh > /var/log/socorro/cron_libraries.log 2>&1

# Daily CSV report for crash-analysis
55 03 * * * socorro /data/bin/cron_daily_reports.sh

# STAGE ONLY - submit crashes to dev
*/3 * * * * socorro /data/socorro/application/scripts/crons/cron_submitter.sh crash-reports-dev.allizom.org 1000 > /dev/null 2>&1

Comment 2

[Robert Helmer [:rhelmer]]

Oh, also we need to clear out crontabber state from earlier attempts.

* rm ~socorro/persistent/crontabbers.json
* delete from crontabbers_state table in psql

I can help with the second.

Comment 3

[Peter Bengtsson [:peterbe]]

(In reply to Robert Helmer [:rhelmer] from comment #2)
> Oh, also we need to clear out crontabber state from earlier attempts.
> 
> * rm ~socorro/persistent/crontabbers.json
> * delete from crontabbers_state table in psql
> 
I'm not sure you can "delete". Only updates are allowed. E.g. `UPDATE crontabbers_states set state = '{}';`

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to Peter Bengtsson [:peterbe] from comment #3)
> (In reply to Robert Helmer [:rhelmer] from comment #2)
> > Oh, also we need to clear out crontabber state from earlier attempts.
> > 
> > * rm ~socorro/persistent/crontabbers.json
> > * delete from crontabbers_state table in psql
> > 
> I'm not sure you can "delete". Only updates are allowed. E.g. `UPDATE
> crontabbers_states set state = '{}';`

Yeah did so, sorry for being sloppy there.

Comment 5

[Robert Helmer [:rhelmer]]

Also, please make sure puppet copies these three files (contain db auth info etc)

/home/rhelmer/bug821930/*.ini -> /etc/socorro/

Comment 6

[Jake Maul [:jakem]]

New crontab added to puppet, 3 new ini files added to puppet, json file removed. rhelmer handled the crontabbers_state table.

Comment 7

[Robert Helmer [:rhelmer]]

(In reply to Jake Maul [:jakem] from comment #6)
> New crontab added to puppet, 3 new ini files added to puppet, json file
> removed. rhelmer handled the crontabbers_state table.

I don't see the three .ini files in socorroadm:/etc/socorro/ any ideas where those got to? :)

Comment 8

[Jake Maul [:jakem]]

Arg.... turns out /etc/socorro isn't recursively managed by puppet... only individual files within. Rather than sort out why this is, I've added explicit resources for these 3 files.

Should be visible within 30 min.