651201 - ensure that cron jobs email on error

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Robert Helmer [:rhelmer]]

Attachment: if pyjob exits non-zero, print info to stdout so crond will email

It's come to my attention that cron jobs aren't always sending us email on error. We are setting the exit code non-zero, but in many cases there is no output.

We are currently redirecting both stderr and stdout to a log file; what we really want is to ensure that the python cron jobs are using stderr appropriately and only log stdout (although this makes reading the logs somewhat painful, we might want to look into putting it in both places).

This will take a little while to test, so I propose for 1.7.7 we print a generic "Cron job (scriptname) exited non-zero: (exit code)"

Comment 1

[Robert Helmer [:rhelmer]]

Got jabba to take a look, and tested using my account on stage:

Committed revision 3069.

Comment 2

[Robert Helmer [:rhelmer]]

Just got an email for a cron job known to be failing (bug 629054):

Subject: Cron <socorro@admin1> /data/socorro/application/scripts/crons/cron_signatures.sh
Cron job startSignatures exited non-zero: 1

Comment 3

[Robert Helmer [:rhelmer]]

(In reply to comment #0)
> We are currently redirecting both stderr and stdout to a log file; what we
> really want is to ensure that the python cron jobs are using stderr
> appropriately and only log stdout (although this makes reading the logs
> somewhat painful, we might want to look into putting it in both places).
> 
> This will take a little while to test, so I propose for 1.7.7 we print a
> generic "Cron job (scriptname) exited non-zero: (exit code)"

Filed bug 651201 to follow up on the above.

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to comment #3)
> (In reply to comment #0)
> > We are currently redirecting both stderr and stdout to a log file; what we
> > really want is to ensure that the python cron jobs are using stderr
> > appropriately and only log stdout (although this makes reading the logs
> > somewhat painful, we might want to look into putting it in both places).
> > 
> > This will take a little while to test, so I propose for 1.7.7 we print a
> > generic "Cron job (scriptname) exited non-zero: (exit code)"
> 
> Filed bug 651201 to follow up on the above.

Sorry meant bug 651652 here.