[crontabber] smarter re-order to prevent excessive delays.

RESOLVED FIXED

Status

RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: peterbe, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

6 years ago
Suppose you have these three apps that depend on each other:

 A -> B -> C

I.e. C depends on B. B depends on A. 
Also, suppose they're entered in the jobs list like this:

    socorro.cron.jobs.C|1h
    socorro.cron.jobs.B|1h
    socorro.cron.jobs.A|1h

What crontabber does is that it loops over that list one at a time (starting from the top) and attempts each one. 
Now, imagine this scenario starting at 12:00

@ 12:00
  1. C: run because B succeeded at 11:00
  2. B: run because A succeeded at 11:00
  3. A: run ...BUT ALL OF A SUDDEN IT FAILS...

@ 13:00
  1. C: run because B succeeded at 12:00
  2. B: can't run because A failed at 12:00
  3. A: run, again, and this time it works (phew!)

@ 14:00
  1. C: can't run because B couldn't run at 13:00
  2. B: run because A succeeded at 13:00 (back to normal)
  3. A: run perfectly fine
  
@ 15:00
  1. C: run because B succeed at 14:00
  2. B: run because A succeed at 14:00
  3. A: run perfectly fine

and now the system is back to normal. 

But suppose we the correct order. Instead of C,B,A we order them according to their dependencies which is: A,B,C. Then it will look like this:

@ 12:00
  1. A: run ...BUT ALL OF A SUDDEN IT FAILS...
  2. B: can't run because A failed at 12:00
  3. C: can't run because B failed at 12:00

@ 13:00 
  1. A: run, again, and this time it works (phew!)
  2. B: run because A succeeded at 13:00 (back to normal)
  3. C: run because B succeeded at 13:00 (back to normal)

See! If ordered correctly, it might not have to wait a whole cycle for its parents to get back into shape. 
In the first case, job B and C had to wait a whole cycle each until they could be re-attempted which could be devastating for daily jobs. 

What we'll do is take our list of jobs we have and at runtime make sure the order is perfect. Since we have much more nested/deeper dependencies it could be a really severe problem. 

(Thanks Lonnen for helping to figure out this flaw)
(Reporter)

Updated

6 years ago
Blocks: 850782
(Reporter)

Updated

6 years ago
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.