854012 - [crontabber] re-try failing jobs much sooner

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Peter Bengtsson [:peterbe]]

If a daily crontabber app fails, which is perfectly normal, you have to wait a whole cycle for it to try again. This can be very damaging for daily jobs.

A better way would be if you can instead re-try 5 minutes after it failed instead of the usual 1 day or 1 hour or whatever the frequency is configured to.

Comment 1

[Peter Bengtsson [:peterbe]]

FYI, I've run an extensive experiment on my local system. With it's own jobs, .json  and .ini file. What the jobs do is that they write their date that they're called with to a log file using `.strftime('%Y-%m-%d %H')`. And before it does this, it checks that it hasn't been written before. This is to simulate what the stored procedures do in that they complain if given the same date twice. 

After running this for many days with injected random errors. All backfills worked all the time. 

All I need to do now with this code is to encapsulate this to a comprehensive (unit) test.

Comment 2

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/57a1149c5e5c085d78b30f8aeab7552ed3ca90c3
fixes bug 854012 - re-try failing crontabber apps sooner, r=rhelmer