Regenerate aggregates that should have run overnight

RESOLVED FIXED

Status

Socorro
General
RESOLVED FIXED
6 years ago
3 years ago

People

(Reporter: laura, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

6 years ago
HBase had an outage overnight and we need to generate aggregates that were not successfully run.
(Reporter)

Comment 1

6 years ago
Not yet sure what specifically we need to re-do, but at least anything involving ADU, so daily and aggregate cron jobs.  Josh, Rob: can one of you take a look at the DB and see if there's anything else we need to rerun?  Once we know exactly what needs doing we can put this in the IT queue.
See Also: → bug 694294
It looks like the newtcbs cron job failed on the postgres function "update_adu":
InternalError: raw_adu not updated for 2011-10-12

I think the only thing that needs to be done is to run the following:

select update_adu('2011-10-12')
select update_daily_crashes('2011-10-12')

Everything else ran ok, and since update_adu caught the problem and aborted we don't need to do backfilling AFAICT.
Rob,
update_adu('2011-10-12') ran fine, but update_daily_crashes('2011-10-12') returned with the message that it was already run.

breakpad=# select update_adu('2011-10-12');
 update_adu 
------------
 t
(1 row)

breakpad=# select update_daily_crashes('2011-10-12');
ERROR:  Daily crashes appears to have already been run for 2011-10-12.  If you want to run it again, please use backfill_daily_crashes()
(In reply to Matt Pressman [:mpressman] from comment #3)
> Rob,
> update_adu('2011-10-12') ran fine, but update_daily_crashes('2011-10-12')
> returned with the message that it was already run.
> 
> breakpad=# select update_adu('2011-10-12');
>  update_adu 
> ------------
>  t
> (1 row)
> 
> breakpad=# select update_daily_crashes('2011-10-12');
> ERROR:  Daily crashes appears to have already been run for 2011-10-12.  If
> you want to run it again, please use backfill_daily_crashes()

Ah right you are - looks like update_daily_crashes does not depend on update_adu in the cronjob, I think it probably should though. 

Josh can you comment here? Do we need to backfill? Should update_daily_crashes depend on update_adu?
daily_crashes is not dependent on any data from raw_adu.  So I don't see any reason why it shouldn't run even if the ADU data isn't there.

In other words, if the graphs are still broken, we have some completely other problem.
(Assignee)

Updated

6 years ago
Component: Socorro → General
Product: Webtools → Socorro
This has been resolved with a more stable daily reports system.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.