677839 - rebuild TCBS for 2011-08-08 to now

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Robert Helmer [:rhelmer]]

Due to processing problems yesterday, we need to rebuild TCBS to correctly reflect crash stats.

Comment 1

[Robert Helmer [:rhelmer]]

going to run:

DELETE FROM top_crashes_by_signature WHERE window_end > '2011-08-08'

Comment 2

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #1)
> going to run:
> 
> DELETE FROM top_crashes_by_signature WHERE window_end > '2011-08-08'

breakpad=> DELETE FROM top_crashes_by_signature WHERE window_end > '2011-08-08';
DELETE 185130

Comment 3

[Robert Helmer [:rhelmer]]

TCBS script runs at 2 minutes past the hour, I'll verify this once it's run and repopulated (probably 1-2 hours).

Wrote a note on the status blog:
http://blog.mozilla.com/socorro/2011/08/09/crash-stats-top-crasher-reports-unavailable-for-2-3-hours/

Comment 4

[Robert Helmer [:rhelmer]]

Looks like TCBS should be caught up now:

2011-08-09 22:10:59,374 INFO - Successfully processed 333880 items in 537.58 seconds
2011-08-09 22:10:59,375 INFO - done.

Kairo, are you able to confirm that the reports/graphs/etc look more reasonable now? If so, please mark verified or reopen. Thanks!

Comment 5

[Robert Kaiser]

(In reply to Robert Helmer [:rhelmer] from comment #4)
> Kairo, are you able to confirm that the reports/graphs/etc look more
> reasonable now? If so, please mark verified or reopen. Thanks!

https://crash-stats.mozilla.com/products/Firefox still shows a significant dip for August 8, i.e. there are values in there for this day that are incorrect.

Newer numbers (August 9) are OK again, though.

Comment 6

[Robert Helmer [:rhelmer]]

(In reply to Robert Kaiser (:[e-mail]) from comment #5)
> (In reply to Robert Helmer [:rhelmer] from comment #4)
> > Kairo, are you able to confirm that the reports/graphs/etc look more
> > reasonable now? If so, please mark verified or reopen. Thanks!
> 
> https://crash-stats.mozilla.com/products/Firefox still shows a significant
> dip for August 8, i.e. there are values in there for this day that are
> incorrect.
> 
> Newer numbers (August 9) are OK again, though.

OK tracked this down to the daily_crashes table, which gets populated nightly from TCBS and ADU tables, and is used for the graphs on the front page and the graphs/table on the "daily" page.

From my read of the script which generates this[1], we should be able to simply remove the last day's worth of data and rerun it. Making a backup first just in case [2] (although I doubt it will be useful).

[1] http://socorro.googlecode.com/svn/branches/releases/2.1/socorro/cron/daily_crash.py
[2] psql -U $databaseUserName -h $databaseHost $databaseName -c "copy (select * from daily_crashes where adu_day >= '2011-08-08') to stdout;" > daily_crashes.out

Comment 7

[Robert Helmer [:rhelmer]]

Once I remove data from this table, we'll need to rerun:

/data/socorro/application/scripts/crons/cron_daily_crashes.sh

Should be done in screen, no arguments needed (logging the output would be nice).

Comment 8

[Robert Helmer [:rhelmer]]

breakpad=> select count(*) from daily_crashes where adu_day >= '2011-08-08';
 count 
-------
   451
(1 row)

breakpad=> delete from daily_crashes where adu_day >= '2011-08-08';
DELETE 451

Comment 9

[Robert Helmer [:rhelmer]]

jabba reran daily_crashes and restarted memcached, looks better now.