851184 - [crontabber] smarter re-order to prevent excessive delays.

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Peter Bengtsson [:peterbe]]

Suppose you have these three apps that depend on each other:

 A -> B -> C

I.e. C depends on B. B depends on A. 
Also, suppose they're entered in the jobs list like this:

    socorro.cron.jobs.C|1h
    socorro.cron.jobs.B|1h
    socorro.cron.jobs.A|1h

What crontabber does is that it loops over that list one at a time (starting from the top) and attempts each one. 
Now, imagine this scenario starting at 12:00

@ 12:00
  1. C: run because B succeeded at 11:00
  2. B: run because A succeeded at 11:00
  3. A: run ...BUT ALL OF A SUDDEN IT FAILS...

@ 13:00
  1. C: run because B succeeded at 12:00
  2. B: can't run because A failed at 12:00
  3. A: run, again, and this time it works (phew!)

@ 14:00
  1. C: can't run because B couldn't run at 13:00
  2. B: run because A succeeded at 13:00 (back to normal)
  3. A: run perfectly fine
  
@ 15:00
  1. C: run because B succeed at 14:00
  2. B: run because A succeed at 14:00
  3. A: run perfectly fine

and now the system is back to normal. 

But suppose we the correct order. Instead of C,B,A we order them according to their dependencies which is: A,B,C. Then it will look like this:

@ 12:00
  1. A: run ...BUT ALL OF A SUDDEN IT FAILS...
  2. B: can't run because A failed at 12:00
  3. C: can't run because B failed at 12:00

@ 13:00 
  1. A: run, again, and this time it works (phew!)
  2. B: run because A succeeded at 13:00 (back to normal)
  3. C: run because B succeeded at 13:00 (back to normal)

See! If ordered correctly, it might not have to wait a whole cycle for its parents to get back into shape. 
In the first case, job B and C had to wait a whole cycle each until they could be re-attempted which could be devastating for daily jobs. 

What we'll do is take our list of jobs we have and at runtime make sure the order is perfect. Since we have much more nested/deeper dependencies it could be a really severe problem. 

(Thanks Lonnen for helping to figure out this flaw)

Comment 1

[Peter Bengtsson [:peterbe]]

Pull request here: https://github.com/mozilla/socorro/pull/1141

Comment 2

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/b24f159f88db0cbfb9dbcfd7f8a9785e6386eac4
bug 851184 - job_list smarter order

https://github.com/mozilla/socorro/commit/2576c06e882ec624b876bb8bbd7a0cfec2540d93
Merge pull request #1141 from peterbe/bug851184-job_list-smarter-order

bug 851184 - job_list smarter order