Closed Bug 882381 Opened 11 years ago Closed 11 years ago

Crontabber failed to self heal after late ADU push

Categories

(Socorro :: Backend, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 880378

People

(Reporter: laura, Assigned: selenamarie)

Details

The ADU push came in late from Metrics for June 11. crontabber did not self heal because it thought the job had succeeded. From the log:
https://gist.github.com/selenamarie/bcd060607b6d3dc402b0

Would be good to know why this broke, and fix it.
rhelmer helped me find out why. 
 
It's right there:

  2013-06-12 03:07:21,875 INFO - MainThread - Notices from calling update_adu([datetime.date(2013, 6, 11)]): ['NOTICE:  raw_adu not updated for 2013-06-11\n']

raw_adu is outside crontabber. So when fed the date 2013-06-11 it should have raised an ERROR. Not a NOTICE. 

The reason for using NOTICE (instead of ERROR) is so that crontabber doesn't get stuck when manual backfilling has happened. 

I suspect the s/ERROR/NOTICE/ went too far. 

Next action::

1. Re-assign to selena

2. Run manual backfill. 

3. Re-audit that stored procedure (and maybe more) and make it so that it ONLY does a `RAISE NOTICE` if it's started with the same date. 

Selena, how do you want to deal with this in terms of filing new bugs? Or perhaps edit the title of this bug?
Assignee: peterbe → sdeckelmann
I filed bug 880378 last week on the same problem, I guess those are duplicates of each other.
(In reply to Peter Bengtsson [:peterbe] from comment #1)
> rhelmer helped me find out why. 
>  
> It's right there:
> 
>   2013-06-12 03:07:21,875 INFO - MainThread - Notices from calling
> update_adu([datetime.date(2013, 6, 11)]): ['NOTICE:  raw_adu not updated for
> 2013-06-11\n']
> 
> raw_adu is outside crontabber. So when fed the date 2013-06-11 it should
> have raised an ERROR. Not a NOTICE. 
> 
> The reason for using NOTICE (instead of ERROR) is so that crontabber doesn't
> get stuck when manual backfilling has happened. 
> 
> I suspect the s/ERROR/NOTICE/ went too far. 
> 
> Next action::
> 
> 1. Re-assign to selena
> 
> 2. Run manual backfill. 
> 
> 3. Re-audit that stored procedure (and maybe more) and make it so that it
> ONLY does a `RAISE NOTICE` if it's started with the same date. 
> 
> Selena, how do you want to deal with this in terms of filing new bugs? Or
> perhaps edit the title of this bug?

ETA is Q3 on this if I am going to do this work, unless there is goal work that we'd like to supercede. Sorry I wasn't more clear about the state of my backlog in IRC.
Target Milestone: --- → 53
Depends on: 880378
Status: NEW → RESOLVED
Closed: 11 years ago
No longer depends on: 880378
Resolution: --- → DUPLICATE
Target Milestone: 53 → ---
You need to log in before you can comment on or make changes to this bug.