44 bytes, text/x-github-pull-request
|Details | Review | Splinter Review|
Let's extend https://crash-stats.allizom.org/monitoring/crontabber/ so that it fails if crontabber hasn't run for a while. What happened in https://bugzilla.mozilla.org/show_bug.cgi?id=1214016 was that due to a config error, starting crontabber every 5 min from cron caused a full on python exception every time. The script /data/socorro/application/scripts/crons/crontabber.sh failed and failed and failed and we did not notice.
We could do what https://crash-stats.allizom.org/monitoring/crontabber/ does which is to talk directly to the crontabber state table in PG. It could check the max date on the last_run date and if that's greater than a certain number we should call it a critical. Alternatively we could extend /data/socorro/application/scripts/crons/crontabber.sh so that it spits out a file or something if the exit code is !0. Or looking at logging? Preferences? Ideas?
Summary: Monitor if crontabber is failing to run → Monitor if crontabber is failing to start at all
What do you think Lars?
For the record, I'm very much in favor of checking the max() date of last_run on the state across all jobs. The only question is, what should the limit be? 24 hours?
Created attachment 8673158 [details] [review] Link to Github pull-request: https://github.com/mozilla/socorro/pull/3033
Commits pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/11b5245de0c70d6f8178bb0f166ac0df8e02538e fixes bug 1214023 - Monitor if crontabber is running, r=AdrianGaudebert https://github.com/mozilla/socorro/commit/49ada687df52119d108fe2583739ce77d6779714 Merge pull request #3033 from peterbe/bug-1214023-monitor-if-crontabber-is-failing-to-start-at-all fixes bug 1214023 - Monitor if crontabber is failing to start at all
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
This broke stage https://errormill.mozilla.org/webtools/socorro-stage/group/400951/
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Created attachment 8673844 [details] [review] Link to Github pull-request: https://github.com/mozilla/socorro/pull/3037
Attachment #8673158 - Attachment is obsolete: true
Commits pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/6edb2225196bfed1228aa47ab0b674ac65eca9e7 fixes bug 1214023 - crontabber monitoring on date comparison https://github.com/mozilla/socorro/commit/3732d17fdadf9c543c9f62d72d2d0fd5acf65b19 Merge pull request #3037 from peterbe/bug-1214023-crontabber-monitoring-on-date-comparison fixes bug 1214023 - crontabber monitoring on date comparison
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago → 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.