413551 - nightly update script should only email on error

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Robert Helmer [:rhelmer]]

Right now the build-announce list is filled with unhelpful informative messages from the cronjob running nightly updates (runs and sends an email every 5 minutes). This script should instead only send an alert on error, and should send it to somewhere that people actually look.

Also, we should use nagios to make sure that the script is actually running, the only benefit of the current system is that we know for sure that it's running; it would be better to have a script actually checking that the queue is being processed.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #0)
> Right now the build-announce list is filled with unhelpful informative messages
> from the cronjob running nightly updates (runs and sends an email every 5
> minutes). This script should instead only send an alert on error, and should
> send it to somewhere that people actually look.

Maybe send errors (and only errors) to build@m.o, like nagios currently does?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I can get this as part of other changes to nightly update generation.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Only email on non-zero exit status

time passes the exit status on, so this is pretty straightforward.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: ... and don't die if we're still running from last time

Converts the die call to an exit 0, since we don't need to know if the run that started 5 minutes is still going.

Comment 5

[Robert Helmer [:rhelmer]]

Comment on attachment 301291 [details] [diff] [review]
... and don't die if we're still running from last time

>Index: patch-packager-cron.sh
>===================================================================
>RCS file: /mofo/release/patcher/patch-packager-cron.sh,v
>retrieving revision 1.3
>diff -u -u -r1.3 patch-packager-cron.sh
>--- patch-packager-cron.sh      31 Jan 2008 14:25:12 -0000      1.3
>+++ patch-packager-cron.sh      4 Feb 2008 12:34:59 -0000
>@@ -8,11 +8,14 @@
> 
> echo "Elapsed time:" > $TIME_FILE
> ( /usr/bin/time -f %E -a -o $TIME_FILE /builds/nightly-partial-generation/app/patch-packager.pl ) >& $OUTPUT_FILE
>+EXIT_STATUS=$?
> 
> echo >> $TIME_FILE
>+echo "Exit status: $EXIT_STATUS" >> $TIME_FILE
>+echo >> $TIME_FILE
> echo "Program output:" >> $TIME_FILE
> 
>-if [ -s $OUTPUT_FILE ]; then
>+if [ $EXIT_STATUS -ne 0 ]; then
>     cat $TIME_FILE $OUTPUT_FILE | \
>         mail -s "AUS2 Report - $DATE" $RCPT
> fi
>Index: patch-packager.pl
>===================================================================
>RCS file: /mofo/release/patcher/patch-packager.pl,v
>retrieving revision 1.15
>diff -u -u -r1.15 patch-packager.pl
>--- patch-packager.pl   1 Oct 2007 18:32:58 -0000       1.15
>+++ patch-packager.pl   4 Feb 2008 16:36:22 -0000
>@@ -66,7 +66,8 @@
>         printf("Found stale PID file... overwriting it.\n");
>         system("echo $our_pid > $pid_file");
>     } else {
>-        die("Patch packager already running with PID $their_pid!\n");
>+        print "Patch packager already running with PID $their_pid! Exiting.\n";
>+        exit 0;
>     }
> }

This will send us an email every time this happens because of the print statement won't it?

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #5)
> This will send us an email every time this happens because of the print
> statement won't it? 

No - it surely adds to the existing file, but we'll throw that away because the exit status is 0. 

Do you think it's worth logging everything on the machine (and using logrotate to control size), while only mailing on error ?

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Checking in patch-packager-cron.sh;
/mofo/release/patcher/patch-packager-cron.sh,v  <--  patch-packager-cron.sh
new revision: 1.4; previous revision: 1.3
done
Checking in patch-packager.pl;
/mofo/release/patcher/patch-packager.pl,v  <--  patch-packager.pl
new revision: 1.16; previous revision: 1.15
done