Closed Bug 849212 Opened 11 years ago Closed 11 years ago

Backfill missing Socorro aggregation data for 2013-03-07

Categories

(Data & BI Services Team :: DB: MySQL, task)

Product:
Data & BI Services Team
Component:
DB: MySQL
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: selenamarie, Assigned: selenamarie)

References

Details

+++ This bug was initially created as a clone of Bug #848947 +++

+++ This bug was initially created as a clone of Bug #848370 +++

Aggregation data on Socorro prod for the 7th is missing.

I checked raw_adu and ADU had come in this morning at 4:30am UTC.
I'm not sure what this message is about (will check in a bit) because ADU came in: 

breakpad=# select backfill_matviews('2013-03-07','2013-03-07');
INFO:  last day of backfill period is after final day of ADU.  adjusting last day to 2013-03-06
INFO:  backfilling reports_clean
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-03-07 00:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-03-07 06:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-03-07 12:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-03-07 18:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
breakpad=# select backfill_matviews('2013-03-07','2013-03-07');
INFO:  last day of backfill period is after final day of ADU.  adjusting last day to 2013-03-06
INFO:  backfilling reports_clean
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-03-07 00:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-03-07 06:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-03-07 12:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-03-07 18:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
INFO:  rank_compare
INFO:  explosiveness (slow)
NOTICE:  table "crash_madu" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS crash_madu"
PL/pgSQL function backfill_explosiveness(date) line 5 at SQL statement
SQL statement "SELECT backfill_explosiveness(thisday)"
PL/pgSQL function backfill_matviews(date,date,boolean) line 87 at PERFORM
NOTICE:  table "xtab_mult" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS xtab_mult"
PL/pgSQL function backfill_explosiveness(date) line 6 at SQL statement
SQL statement "SELECT backfill_explosiveness(thisday)"
PL/pgSQL function backfill_matviews(date,date,boolean) line 87 at PERFORM
NOTICE:  table "crash_xtab" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS crash_xtab"
PL/pgSQL function backfill_explosiveness(date) line 7 at SQL statement
SQL statement "SELECT backfill_explosiveness(thisday)"
PL/pgSQL function backfill_matviews(date,date,boolean) line 87 at PERFORM
NOTICE:  table "explosive_oneday" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS explosive_oneday"
PL/pgSQL function backfill_explosiveness(date) line 8 at SQL statement
SQL statement "SELECT backfill_explosiveness(thisday)"
PL/pgSQL function backfill_matviews(date,date,boolean) line 87 at PERFORM
NOTICE:  table "explosive_threeday" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS explosive_threeday"
PL/pgSQL function backfill_explosiveness(date) line 9 at SQL statement
SQL statement "SELECT backfill_explosiveness(thisday)"
PL/pgSQL function backfill_matviews(date,date,boolean) line 87 at PERFORM
INFO:  correlations
 backfill_matviews 
-------------------
 t
(1 row)

Time: 828027.275 ms


Hmmm none of the reports ran.
Summary: Backfill missing Socorro aggregation data for 2013-03-06 → Backfill missing Socorro aggregation data for 2013-03-07
Upon further investigation, we see that crontabber saw that backfill needed to be done for the missing data from the 6th (even though *we* ran backfill on our own). 

However, when it reached BuildADUCronApp, it only ran the job for the 6th. Lars is now researching whether we can do a one-off job to get the BuildADUCronApp and its dependent jobs to run, or if we should manually run those.
Lars reports that the command line to re-run this is: 

07:33 <        lars> | python /data/socorro/application/socorro/cron/crontabber.py --admin.conf=/etc/socorro/crontabber.ini -j socorro.cron.jobs.matviews.BuildADUCronApp
07:33 <        lars> | with PYTHONPATH=/data/socorro/application/:/data/socorro/application/scripts:/data/socorro/thirdparty/
We cannot force a backfill to run at an unspecified time through crontabber.

Running the backfill manually again.
The underlying problem was missing ADU. I misinterpreted the dates, which spawned this ticket: https://bugzilla.mozilla.org/show_bug.cgi?id=849250

Wheeee!
breakpad=# select backfill_matviews('2013-03-07','2013-03-07');
INFO:  backfilling reports_clean
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-03-07 00:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-03-07 06:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-03-07 12:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-03-07 18:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
INFO:  backfilling other matviews for 2013-03-07
INFO:  adu
INFO:  signatures
INFO:  tcbs
NOTICE:  table "new_tcbs" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS new_tcbs"
PL/pgSQL function backfill_matviews(date,date,boolean) line 66 at SQL statement
INFO:  crashes by user
INFO:  home page graph
INFO:  hang report (slow)
INFO:  nightly builds
INFO:  rank_compare
INFO:  explosiveness (slow)
NOTICE:  table "crash_madu" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS crash_madu"
PL/pgSQL function backfill_explosiveness(date) line 5 at SQL statement
SQL statement "SELECT backfill_explosiveness(thisday)"
PL/pgSQL function backfill_matviews(date,date,boolean) line 87 at PERFORM
NOTICE:  table "xtab_mult" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS xtab_mult"
PL/pgSQL function backfill_explosiveness(date) line 6 at SQL statement
SQL statement "SELECT backfill_explosiveness(thisday)"
PL/pgSQL function backfill_matviews(date,date,boolean) line 87 at PERFORM
NOTICE:  table "crash_xtab" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS crash_xtab"
PL/pgSQL function backfill_explosiveness(date) line 7 at SQL statement
SQL statement "SELECT backfill_explosiveness(thisday)"
PL/pgSQL function backfill_matviews(date,date,boolean) line 87 at PERFORM
NOTICE:  table "explosive_oneday" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS explosive_oneday"
PL/pgSQL function backfill_explosiveness(date) line 8 at SQL statement
SQL statement "SELECT backfill_explosiveness(thisday)"
PL/pgSQL function backfill_matviews(date,date,boolean) line 87 at PERFORM
NOTICE:  table "explosive_threeday" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS explosive_threeday"
PL/pgSQL function backfill_explosiveness(date) line 9 at SQL statement
SQL statement "SELECT backfill_explosiveness(thisday)"
PL/pgSQL function backfill_matviews(date,date,boolean) line 87 at PERFORM
INFO:  correlations
-[ RECORD 1 ]-----+--
backfill_matviews | t
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Assignee: server-ops-database → sdeckelmann
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.