965244 - release 72 has broken the cron submitter

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[K Lars Lohn [:lars] [:klohn]]

at 6:42, the cron submitter, the process that submits crashes from production to staging broke.  The new configuration in svn eliminates some old files.  Strangely, the cron submitter relies on the existence of "/etc/socorro/socorro-monitor.conf".   This needs to be fixed or the new method of moving crashes from prod to stage implemented immediately.

Comment 1

[K Lars Lohn [:lars] [:klohn]]

this problem can be solved with the new submitter_app and configuration alone.  This would deprecate the .../scripts/crons/cron_submitter.sh file entirely.

1) create a cron_submitter.ini file for submitter_app.py that will select 1000 crash_ids from PG, pull them from hbase and submit to staging.  [lars has this done]

2) create a cron_submitter_secrets.ini file for the postgres login information  [lars has this done]

3) commit these ini files to prod svn

4) delete the cron_submitter.sh cron job on the admin box

5) create a new cron job with the same timing as the old one, with 'flock' for locking:
    python /data/socorro/application/socorro/collector/submitter_app.py --admin.conf=config/cron_submitter.ini

we're done.

Comment 2

[Brandon Burton [:solarce]]

This all sounds good, proceed with step 3 and I'll commit these changes for steps 4,5

-> % svn diff
Index: prod/etc-crond/socorro
===================================================================
--- prod/etc-crond/socorro	(revision 81448)
+++ prod/etc-crond/socorro	(working copy)
@@ -19,4 +19,4 @@
 0 * * * * socorro /data/socorro/application/scripts/crons/cron_fixbrokendumps.sh

 # PROD ONLY - submit crashes to stage
-*/4 * * * * socorro /data/socorro/application/scripts/crons/cron_submitter.sh crash-reports.allizom.org 1000
+*/4 * * * * socorro /usr/bin/flock -w 30 /var/lock/socorro-cron-stagecrashsubmitter.lock python /data/socorro/application/socorro/collector/submitter_app.py --admin.conf=config/cron_submitter.ini

Comment 3

[Brandon Burton [:solarce]]

-> % svn ci -m "new cron submitter cronjob for socorro prod, bug 965244"
Sending        files/prod/etc-crond/socorro
Transmitting file data .
Committed revision 81470.

Comment 4

[Brandon Burton [:solarce]]

Working on this in https://github.com/mozilla/socorro/pull/1835

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/bf1ea88fc87c7c09a01fc7bf0c69102f3cd88999
bug 965244 - updating cron_submitter.sh to use submitter_app.py, still needs shell script to invoke ENV stuff so cron is happy

https://github.com/mozilla/socorro/commit/b669c28e0d669205f68c05996b3b5ea005ddf3e0
bug 965244 - determined socorro-monitor.conf isn't needed

https://github.com/mozilla/socorro/commit/2ed19225260e3db1ff2cdf997123527137f4c365
Merge pull request #1835 from solarce/master

bug 965244 - updating cron_submitter.sh to use submitter_app.py, still n...