1214023 - Monitor if crontabber is failing to start at all

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Peter Bengtsson [:peterbe]]

Let's extend https://crash-stats.allizom.org/monitoring/crontabber/ so that it fails if crontabber hasn't run for a while. 

What happened in https://bugzilla.mozilla.org/show_bug.cgi?id=1214016 was that due to a config error, starting crontabber every 5 min from cron caused a full on python exception every time. 

The script /data/socorro/application/scripts/crons/crontabber.sh failed and failed and failed and we did not notice.

Comment 1

[Peter Bengtsson [:peterbe]]

We could do what https://crash-stats.allizom.org/monitoring/crontabber/ does which is to talk directly to the crontabber state table in PG. It could check the max date on the last_run date and if that's greater than a certain number we should call it a critical. 

Alternatively we could extend /data/socorro/application/scripts/crons/crontabber.sh so that it spits out a file or something if the exit code is !0. 

Or looking at logging?

Preferences? Ideas?

Comment 2

[Peter Bengtsson [:peterbe]]

What do you think Lars?

Comment 3

[Peter Bengtsson [:peterbe]]

For the record, I'm very much in favor of checking the max() date of last_run on the state across all jobs. The only question is, what should the limit be? 24 hours?

Comment 4

[Peter Bengtsson [:peterbe]]

Attachment: Link to Github pull-request: https://github.com/mozilla/socorro/pull/3033

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/11b5245de0c70d6f8178bb0f166ac0df8e02538e
fixes bug 1214023 - Monitor if crontabber is running, r=AdrianGaudebert

https://github.com/mozilla/socorro/commit/49ada687df52119d108fe2583739ce77d6779714
Merge pull request #3033 from peterbe/bug-1214023-monitor-if-crontabber-is-failing-to-start-at-all

fixes bug 1214023 - Monitor if crontabber is failing to start at all

Comment 6

[Peter Bengtsson [:peterbe]]

This broke stage 
https://errormill.mozilla.org/webtools/socorro-stage/group/400951/

Comment 7

[Peter Bengtsson [:peterbe]]

Attachment: Link to Github pull-request: https://github.com/mozilla/socorro/pull/3037

Comment 8

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/6edb2225196bfed1228aa47ab0b674ac65eca9e7
fixes bug 1214023 - crontabber monitoring on date comparison

https://github.com/mozilla/socorro/commit/3732d17fdadf9c543c9f62d72d2d0fd5acf65b19
Merge pull request #3037 from peterbe/bug-1214023-crontabber-monitoring-on-date-comparison

fixes bug 1214023 - crontabber monitoring on date comparison