Currently we're making a check on the nightly report and we don't send it if the difference between the day and d/d-7 is higher than 10%. This is not the best approach, though. We need to replace this with a decent integrity check; Before sending the reports, and after all etl loads, we can check if we're missing any block of data for any of the major sources and, if any chunk of data is missing, send an alert to the metrics team and prevent the report from being sent.
The queries in cm-metricsdb01:/home/dbadmin/sql/histogram_raw.sh are a good start for this. They do a reverse join on date and the raw table and they will show any missing hours. We'd want to further break down by source_type to see if there is any server or cluster of servers that has an unusual traffic pattern.
My thoughts exactly!
Assignee: pedro.alves → deinspanjer
Whiteboard: Provide hook for green/red light to Pedro's team ETA 2010-03-02
Need copy of current nightly report going out to stats-reports@ an hour after the one going to metrics@. Would like both reports to use something like the following query as the primary safety check: SELECT 1 FROM eod_status WHERE utc_date = CURRENT_DATE AND status = 'Successful'
We need one more thing to call this done: We need to have a kill switch to prevent the second report from being e-mailed to stats-reports@mozillacom
Whiteboard: Provide hook for green/red light to Pedro's team ETA 2010-03-02 → Need kill switch as last piece 2010-04-16
Will push to production Monday morning.
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.