852112 - Backfill missing Socorro aggregation data for 2013-03-17

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Robert Kaiser]

*Big sigh*

Aggregation data on Socorro prod for the 17th is missing yet again, so we need to backfill that data for any crash analysis work to be done on it.

Comment 1

[Sheeri Cabral [:sheeri]]

Will work on backfill.

How is this recurring issue going to be resolved?

Comment 2

[Sheeri Cabral [:sheeri]]

backfill begun.

Comment 3

[Robert Kaiser]

(In reply to Sheeri Cabral [:sheeri] from comment #1)
> How is this recurring issue going to be resolved?

Laura, can you and the team come up with a plan there? This is getting quite annoying (6th time in two weeks).

Comment 4

[Robert Kaiser]

FWIW, when my reports ran, they didn't get data from raw_adu (I have one reports fetching directly from there to get accurate per-build data), so I guess this time the cause was another late ADI push from metrics.

Comment 5

[Sheeri Cabral [:sheeri]]

 backfill_matviews 
-------------------
 t
(1 row)

Time: 3703620.376 ms

I have also restarted memcached on the webheads.

(can you let me know the bug that has the plan for trying to resolve this issue so I can cc myself on it?)

Comment 6

[Laura Thomson :laura]

Trying to fix this in 
https://bugzilla.mozilla.org/show_bug.cgi?id=850782

If that's taking too long for you, we could roll back crontabber again.  Thoughts?

Comment 7

[Robert Kaiser]

I can verify that the backfill is done and we have the data, my reports are backfilled now as well. Is there a reason this bug itself is still open?

Comment 8

[Robert Helmer [:rhelmer]]

(In reply to Laura Thomson :laura from comment #6)
> Trying to fix this in 
> https://bugzilla.mozilla.org/show_bug.cgi?id=850782
> 
> If that's taking too long for you, we could roll back crontabber again. 
> Thoughts?

If the problem is late ADI, the old system also needs to have nightly jobs manually backfilled if it's not there in time (we run reports at 10 AM UTC for the previous day, but data for ADI did not come in for 2013-03-17 until 2013-03-18 12:59:11).

Bug 850782 is about building a feature into crontabber to retry more often in this type of scenario, so it can fix it automatically.

Comment 9

[Sheeri Cabral [:sheeri]]

Robert - the bug is still open because there are still open questions about the repeated failure. Feel free to change the subject or un-cc yourself if the extra chatter is distracting.

As for rolling back the crontabber, I'm not sure what exactly rolling it back would mean as a consequence for everything else. If it means too much pain for others, it's not worth it, but having to run backfills several times a week instead of twice a month is definitely frustrating.

Can this be more accurately tested on dev/stage before it goes live?

Comment 10

[Robert Kaiser]

In the case of late ADI pushes, like we had today, rolling back crontabber doesn't help. For that, we need some kind of "retry once the required data actually is there".

Comment 11

[Laura Thomson :laura]

Yeah, as Kairo said, in this case it doesn't help.  The next step is bifold:
1. Find out from Metrics why the ADIs keep coming in late
2. Consider moving the crons later to avoid backfilling.  Of course, this doens't help Kairo do his job.

WRT crontabber, we had it in staging for a long time and did not have errors.  Edge cases of intermittent external failures and their interactions with timing are hard to test.  And in this case we'd be running backfill anyway.

Comment 12

[Robert Kaiser]

(In reply to Laura Thomson :laura from comment #11)
> 2. Consider moving the crons later to avoid backfilling.  Of course, this
> doens't help Kairo do his job.

Exactly, as anyone here in Europe (like me) already can only start analyzing data in the afternoon, even though the aggregation is for UTC days. Only being able to work on data for the last day towards the end of a normal work day in Europe is surely no solution to make things easier for contributors (paid or not) on this side of the Atlantic.

Comment 13

[Robert Kaiser]

Is there still anything to do here? I just found that this bug is still open.