1581938 - Generate alerts if Heroku scheduler tasks fail to execute

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, enhancement)

Description

[Cameron Dawson [:camd]]

We should get a New Relic alert if this fails, so we keep our database as trim as possible.

Comment 1

[Cameron Dawson [:camd]]

Hey Armen-- Here's the bug you asked for. I tried creating an alert, but didn't have success with it. Thanks for your help!

Comment 2

[Armen [:armenzg]]

Attachment: image.png

I've added the add-on to treeherder-stage and I appended this to the Heroku scheduler:

$ newrelic-admin run-program ./manage.py cycle_data --chunk-size=10000; curl -d s=$? https://nosnch.in/d7c996637e &> /dev/null

It will first alert me and then I will change it to the team.

Comment 3

[Armen [:armenzg]]

I've set up an add-on on Heroku (Dead man snitch) that keeps track that periodic tasks run successfully. This is accomplished by appending to a Heroku scheduler command a cURL call (&& curl https://nosnich.in/c2354d53d). If a periodic task has not run the last two times it will email us.

For now, it will only alert me until I'm satisfied and I change it.

Notice that I use " &&" [1] rather than ";" [2] because I only want to call back home if the script executes with sucess.
If we want more fine-combing (like tracking status codes and logs) we will need Field Agent installed, through its buildpack [3] and we will be able to use Error Notices [4].

[1]

$ cat foo.txt && echo "I will only execute if the previous command has an exit code of 0"
cat: foo.txt: No such file or directory

[2]

$ cat foo.txt; echo "I will execute regardless of the exit code from the previous command"
cat: foo.txt: No such file or directory
I will execute regardless of the exit code from the previous command

[3]

the dms command will be available in your dyno's shell PATH.

[4]

It's now possible (and highly recommended) to send your job's exit status along with your check-in rather than checking in only on success.
If your job runs but errors, we will immediately notify you if something went wrong.

Comment 4

[Armen [:armenzg]]

All reporting correctly.