Closed
Bug 882381
Opened 11 years ago
Closed 11 years ago
Crontabber failed to self heal after late ADU push
Categories
(Socorro :: Backend, task)
Socorro
Backend
Tracking
(Not tracked)
RESOLVED
DUPLICATE
of bug 880378
People
(Reporter: laura, Assigned: selenamarie)
Details
The ADU push came in late from Metrics for June 11. crontabber did not self heal because it thought the job had succeeded. From the log: https://gist.github.com/selenamarie/bcd060607b6d3dc402b0 Would be good to know why this broke, and fix it.
Comment 1•11 years ago
|
||
rhelmer helped me find out why. It's right there: 2013-06-12 03:07:21,875 INFO - MainThread - Notices from calling update_adu([datetime.date(2013, 6, 11)]): ['NOTICE: raw_adu not updated for 2013-06-11\n'] raw_adu is outside crontabber. So when fed the date 2013-06-11 it should have raised an ERROR. Not a NOTICE. The reason for using NOTICE (instead of ERROR) is so that crontabber doesn't get stuck when manual backfilling has happened. I suspect the s/ERROR/NOTICE/ went too far. Next action:: 1. Re-assign to selena 2. Run manual backfill. 3. Re-audit that stored procedure (and maybe more) and make it so that it ONLY does a `RAISE NOTICE` if it's started with the same date. Selena, how do you want to deal with this in terms of filing new bugs? Or perhaps edit the title of this bug?
Assignee: peterbe → sdeckelmann
Comment 2•11 years ago
|
||
I filed bug 880378 last week on the same problem, I guess those are duplicates of each other.
Assignee | ||
Comment 3•11 years ago
|
||
(In reply to Peter Bengtsson [:peterbe] from comment #1) > rhelmer helped me find out why. > > It's right there: > > 2013-06-12 03:07:21,875 INFO - MainThread - Notices from calling > update_adu([datetime.date(2013, 6, 11)]): ['NOTICE: raw_adu not updated for > 2013-06-11\n'] > > raw_adu is outside crontabber. So when fed the date 2013-06-11 it should > have raised an ERROR. Not a NOTICE. > > The reason for using NOTICE (instead of ERROR) is so that crontabber doesn't > get stuck when manual backfilling has happened. > > I suspect the s/ERROR/NOTICE/ went too far. > > Next action:: > > 1. Re-assign to selena > > 2. Run manual backfill. > > 3. Re-audit that stored procedure (and maybe more) and make it so that it > ONLY does a `RAISE NOTICE` if it's started with the same date. > > Selena, how do you want to deal with this in terms of filing new bugs? Or > perhaps edit the title of this bug? ETA is Q3 on this if I am going to do this work, unless there is goal work that we'd like to supercede. Sorry I wasn't more clear about the state of my backlog in IRC.
Comment 4•11 years ago
|
||
This is the Pull Request right: https://github.com/mozilla/socorro/pull/1316
Assignee | ||
Updated•11 years ago
|
Target Milestone: --- → 53
Assignee | ||
Updated•11 years ago
|
Assignee | ||
Updated•11 years ago
|
Target Milestone: 53 → ---
You need to log in
before you can comment on or make changes to this bug.
Description
•