860233 - Aggregations not filled in for April 9

Status: RESOLVED FIXED

(Socorro :: Backend, task)

Description

[Robert Kaiser]

I'm filing this against backend atm, because it feels like crontabber didn't do what it should do.

Yesterday, we did get ADU late:
breakpad=> SELECT max(received_at) FROM raw_adu WHERE date='2013-04-08';
              max              
-------------------------------
 2013-04-09 11:28:11.910446+00
(1 row)

Because of that and our aggressive retry, we did run the daily aggregations job at a later time, but it worked fine (yay).

Today, ADU are on time:
breakpad=> SELECT max(received_at) FROM raw_adu WHERE date='2013-04-09';
              max              
-------------------------------
 2013-04-10 03:29:26.123832+00
(1 row)

The daily aggregations for yesterday haven't been filled in (yet), though.


Is there a problem with crontabber there, e.g. that we might not run the job if the last one we did run was fewer than 24 hours ago?

Comment 1

[Peter Bengtsson [:peterbe]]

Seconds ago I peek at prod's crontabber state and it says:

 * build-adu-matview: 
 ** "last_success": "2013-04-09 10:00:16.365841"
 ** "last_run": "2013-04-10 10:00:01.976561",
 * adu-matview: 
 **  "last_success": "2013-04-09 10:00:48.600524",
 **  "last_run": "2013-04-10 10:00:02.913895",

There are no current errors in the state, for example:

 * home-page-graph-matview: 
 ** "last_success": "2013-04-09 10:00:47.507316",
 ** "last_run": "2013-04-10 10:00:02.891570",

So, apparently it's behind by 1 day. 

And I think I know why: https://bugzilla.mozilla.org/show_bug.cgi?id=857100
Basically, when it comes around to 24 hours later, it realizes that only 23h 59min 45seconds have lapsed and therefore it doesn't backfill for the past 1 day. 

The code for #857100 has already landed on master but I fear it hasn't been pushed out to stage or prod yet.

Comment 2

[Selena Deckelmann :selenamarie :selena]

breakpad=# select backfill_matviews('2013-04-09', '2013-04-09');INFO:  backfilling reports_clean
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-04-09 00:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-04-09 06:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-04-09 12:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
INFO:  backfilling 06:00:00 of reports_clean starting at 2013-04-09 18:00:00+00
CONTEXT:  SQL statement "SELECT backfill_reports_clean( first_rc, last_rc )"
PL/pgSQL function backfill_matviews(date,date,boolean) line 52 at PERFORM
INFO:  backfilling other matviews for 2013-04-09
INFO:  adu
INFO:  signatures
INFO:  tcbs
NOTICE:  table "new_tcbs" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS new_tcbs"
PL/pgSQL function backfill_matviews(date,date,boolean) line 66 at SQL statement
INFO:  crashes by user
INFO:  home page graph
INFO:  hang report (slow)
INFO:  nightly builds
INFO:  rank_compare
INFO:  explosiveness (slow)
NOTICE:  table "crash_madu" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS crash_madu"
PL/pgSQL function backfill_explosiveness(date) line 5 at SQL statement
SQL statement "SELECT backfill_explosiveness(thisday)"
PL/pgSQL function backfill_matviews(date,date,boolean) line 87 at PERFORM
NOTICE:  table "xtab_mult" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS xtab_mult"
PL/pgSQL function backfill_explosiveness(date) line 6 at SQL statement
SQL statement "SELECT backfill_explosiveness(thisday)"
PL/pgSQL function backfill_matviews(date,date,boolean) line 87 at PERFORM
NOTICE:  table "crash_xtab" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS crash_xtab"
PL/pgSQL function backfill_explosiveness(date) line 7 at SQL statement
SQL statement "SELECT backfill_explosiveness(thisday)"
PL/pgSQL function backfill_matviews(date,date,boolean) line 87 at PERFORM
NOTICE:  table "explosive_oneday" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS explosive_oneday"
PL/pgSQL function backfill_explosiveness(date) line 8 at SQL statement
SQL statement "SELECT backfill_explosiveness(thisday)"
PL/pgSQL function backfill_matviews(date,date,boolean) line 87 at PERFORM
NOTICE:  table "explosive_threeday" does not exist, skipping
CONTEXT:  SQL statement "DROP TABLE IF EXISTS explosive_threeday"
PL/pgSQL function backfill_explosiveness(date) line 9 at SQL statement
SQL statement "SELECT backfill_explosiveness(thisday)"
PL/pgSQL function backfill_matviews(date,date,boolean) line 87 at PERFORM
INFO:  correlations
 backfill_matviews 
-------------------
 t
(1 row)

Time: 4125512.231 ms

Comment 3

[Robert Kaiser]

Selena, thanks, with this the bug is fixed - the actual issue behind the problem is being tracked in bug 857100 anyhow.