Closed Bug 793885 Opened 13 years ago Closed 12 years ago

Limit dependencies on adu in crontabber

Categories

(Socorro :: Backend, task)

Product:
Socorro
Component:
Backend
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: selenamarie, Assigned: selenamarie)

References

Details

(Whiteboard: [qa-])

Create a cronjob to pull ADU data from metrics instead of the push.
To fill out some more detail here, we were thinking it would be easier if we pulled data from Vertica instead of waiting on a push. This would make monitoring and failure recovery a lot easier on our end. Daniel, can you help us get started on this?
We could do it, but I am not sure it really buys you much. You would still have to schedule it to run after the EOD process, and if the EOD fails, your pull would fail as well, and it would have to either be manually run it after the EOD is cleared up, or it would have to keep retrying. What sort of improvements would you like to make to monitoring and recovery exactly?
What it buys me is that when something like https://bugzilla.mozilla.org/show_bug.cgi?id=805402 happens, I don't have to come and chase somebody down on IRC to push the data to us. We could just script retries. We could also more easily set a monitor on this log, so if we hadn't received ADU data we would know before our users tell us.
That would make sense for anything on your end that has to work with the data once it appears in your adu table, but the most common problem that can happen (as in bug 805402) is that there is some problem with the EOD that generates the numbers in the first place, so if we had a script that tried to run the export query itself instead of as part of the EOD, then it would not get valid data until the EOD succeeded anyway, and we currently have the export set up as the last action on a successful EOD. One thing that AMO has that I would encourage you guys to implement is a nagios check that validates whether your table has the most recent data you expect. If the EOD fails, it is usually the very first thing that Metrics people in Portugal or the east coast USA look at when we start work. There is no way that a failure would be missed. The only thing that might be overlooked is an occasional failure in the export to staging because we expect that to fail occasionally if it happens while staging is down for upgrades or refresh.
Summary: Create python cron for ADU → Limit dependencies on adu in crontabber
Commits pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/4fa730c4a55c3d1ccd74a53df4e499d8b58850f3 Fixes bug 793885 Limit dependencies on ADU in matviews https://github.com/mozilla/socorro/commit/a76a0fdd794558dbfffff0fcdc2de95d388c591c Merge pull request #1477 from selenamarie/bug793885-limit-adu-deps Fixes bug 793885 Limit dependencies on ADU in matviews
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Whiteboard: [qa-]
Assignee: nobody → sdeckelmann
Target Milestone: --- → 59
Depends on: 970406
You need to log in before you can comment on or make changes to this bug.