Closed
Bug 1214023
Opened 9 years ago
Closed 9 years ago
Monitor if crontabber is failing to start at all
Categories
(Socorro :: Infra, task)
Socorro
Infra
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: peterbe, Assigned: peterbe)
Details
Attachments
(1 file, 1 obsolete file)
Let's extend https://crash-stats.allizom.org/monitoring/crontabber/ so that it fails if crontabber hasn't run for a while. What happened in https://bugzilla.mozilla.org/show_bug.cgi?id=1214016 was that due to a config error, starting crontabber every 5 min from cron caused a full on python exception every time. The script /data/socorro/application/scripts/crons/crontabber.sh failed and failed and failed and we did not notice.
Assignee | ||
Comment 1•9 years ago
|
||
We could do what https://crash-stats.allizom.org/monitoring/crontabber/ does which is to talk directly to the crontabber state table in PG. It could check the max date on the last_run date and if that's greater than a certain number we should call it a critical. Alternatively we could extend /data/socorro/application/scripts/crons/crontabber.sh so that it spits out a file or something if the exit code is !0. Or looking at logging? Preferences? Ideas?
Assignee | ||
Updated•9 years ago
|
Summary: Monitor if crontabber is failing to run → Monitor if crontabber is failing to start at all
Assignee | ||
Comment 3•9 years ago
|
||
For the record, I'm very much in favor of checking the max() date of last_run on the state across all jobs. The only question is, what should the limit be? 24 hours?
Assignee | ||
Updated•9 years ago
|
Assignee: nobody → peterbe
Assignee | ||
Comment 4•9 years ago
|
||
Comment 5•9 years ago
|
||
Commits pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/11b5245de0c70d6f8178bb0f166ac0df8e02538e fixes bug 1214023 - Monitor if crontabber is running, r=AdrianGaudebert https://github.com/mozilla/socorro/commit/49ada687df52119d108fe2583739ce77d6779714 Merge pull request #3033 from peterbe/bug-1214023-monitor-if-crontabber-is-failing-to-start-at-all fixes bug 1214023 - Monitor if crontabber is failing to start at all
Updated•9 years ago
|
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 6•9 years ago
|
||
This broke stage https://errormill.mozilla.org/webtools/socorro-stage/group/400951/
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee | ||
Comment 7•9 years ago
|
||
Attachment #8673158 -
Attachment is obsolete: true
Comment 8•9 years ago
|
||
Commits pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/6edb2225196bfed1228aa47ab0b674ac65eca9e7 fixes bug 1214023 - crontabber monitoring on date comparison https://github.com/mozilla/socorro/commit/3732d17fdadf9c543c9f62d72d2d0fd5acf65b19 Merge pull request #3037 from peterbe/bug-1214023-crontabber-monitoring-on-date-comparison fixes bug 1214023 - crontabber monitoring on date comparison
Updated•9 years ago
|
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Updated•8 years ago
|
Flags: needinfo?(lars)
You need to log in
before you can comment on or make changes to this bug.
Description
•