Monitor if crontabber is failing to start at all

RESOLVED FIXED

Status

RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: peterbe, Assigned: peterbe)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 1 obsolete attachment)

(Assignee)

Description

3 years ago
Let's extend https://crash-stats.allizom.org/monitoring/crontabber/ so that it fails if crontabber hasn't run for a while. 

What happened in https://bugzilla.mozilla.org/show_bug.cgi?id=1214016 was that due to a config error, starting crontabber every 5 min from cron caused a full on python exception every time. 

The script /data/socorro/application/scripts/crons/crontabber.sh failed and failed and failed and we did not notice.
(Assignee)

Comment 1

3 years ago
We could do what https://crash-stats.allizom.org/monitoring/crontabber/ does which is to talk directly to the crontabber state table in PG. It could check the max date on the last_run date and if that's greater than a certain number we should call it a critical. 

Alternatively we could extend /data/socorro/application/scripts/crons/crontabber.sh so that it spits out a file or something if the exit code is !0. 

Or looking at logging?

Preferences? Ideas?
(Assignee)

Updated

3 years ago
Summary: Monitor if crontabber is failing to run → Monitor if crontabber is failing to start at all
(Assignee)

Comment 2

3 years ago
What do you think Lars?
Flags: needinfo?(lars)
(Assignee)

Comment 3

3 years ago
For the record, I'm very much in favor of checking the max() date of last_run on the state across all jobs. The only question is, what should the limit be? 24 hours?
(Assignee)

Updated

3 years ago
Assignee: nobody → peterbe

Comment 5

3 years ago
Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/11b5245de0c70d6f8178bb0f166ac0df8e02538e
fixes bug 1214023 - Monitor if crontabber is running, r=AdrianGaudebert

https://github.com/mozilla/socorro/commit/49ada687df52119d108fe2583739ce77d6779714
Merge pull request #3033 from peterbe/bug-1214023-monitor-if-crontabber-is-failing-to-start-at-all

fixes bug 1214023 - Monitor if crontabber is failing to start at all

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Assignee)

Comment 6

3 years ago
This broke stage 
https://errormill.mozilla.org/webtools/socorro-stage/group/400951/
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 7

3 years ago
Created attachment 8673844 [details] [review]
Link to Github pull-request: https://github.com/mozilla/socorro/pull/3037
Attachment #8673158 - Attachment is obsolete: true

Comment 8

3 years ago
Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/6edb2225196bfed1228aa47ab0b674ac65eca9e7
fixes bug 1214023 - crontabber monitoring on date comparison

https://github.com/mozilla/socorro/commit/3732d17fdadf9c543c9f62d72d2d0fd5acf65b19
Merge pull request #3037 from peterbe/bug-1214023-crontabber-monitoring-on-date-comparison

fixes bug 1214023 - crontabber monitoring on date comparison

Updated

3 years ago
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
Flags: needinfo?(lars)
You need to log in before you can comment on or make changes to this bug.