688587 - Missing _all_ ADU information in Socorro for Firefox versions

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Marcia Knous [:marcia]]

I see "0" ADUs for both Firefox 9 and Firefox 7.0b6. KaiRo asked me to cc Daniel. dveditz and jst say the nightly report they get from Metrics is showing the data.

Comment 1

[Daniel Einspanjer [:dre] [:deinspanjer]]

ADU data was not exported for 2011-09-21 until early this morning (it usually happens late at night on that date Pacific time).

The EOD process encountered an error and someone responded to the issue so I thought it was taken care of, but it was not fully re-run.

Comment 2

[Marcia Knous [:marcia]]

Sorry, in my haste to file this bug I forgot to add where I was seeing it:

https://crash-stats.mozilla.com/daily?p=Firefox&v[]=

And in Comment 0 I meant 9.0a1.

Comment 3

[Robert Helmer [:rhelmer]]

jberkus reran the import to pick up the late ADU export, and we're investigating why we weren't alerted (and also adding additional monitoring).

This looks good to me on crash-stats now, please reopen if it does not to you.

Comment 4

[Marcia Knous [:marcia]]

https://crash-stats.mozilla.com/daily?p=Firefox&v[]= - Today smooney and I see 0 ADUs, but only for 9.0a1 and 8.0a2. Did the monitoring pick this up?  Thanks.

Comment 5

[Marcia Knous [:marcia]]

I can create a new bug if you feel that this is not the same issue that happened before.

Comment 6

[Marcia Knous [:marcia]]

https://crash-stats.mozilla.com/daily?p=Firefox&v[]=9.0a1 and https://crash-stats.mozilla.com/daily?p=Firefox&v[]=8.0a2 are the two specific URLs that show no ADUs today.

Comment 7

[Laura Thomson :laura]

Just wanted to comment that we are looking at the issue.  From IRC discussion:

deinspanjer: rhelmer: I am not sure, but I suspect it might have gone through late
deinspanjer: I made a change this morning to make it a little less likely to fail by doing each of the exports serially instead of in parallel.  That will also make it easy for us to figure out if / when it went through.
deinspanjer: This is only a problem because our current cluster is at capacity and IT has been having trouble getting the new cluster on-line. 
rhelmer: KaiRo: sounds like it just wasn't all finished in time when we ran
KaiRo: deinspanjer, rhelmer: maybe we have been late and aggregated on a not-fully-filled set?
rhelmer: KaiRo: I suspect we just need to backfill, I'll need to look at the docs jberkus left since he's not around today

Comment 8

[Laura Thomson :laura]

Regenerating and backfilling data for 2.2.4 picked up the late ADUs, and they are now visible at the two URLs that Marcia listed.

The new code makes it easier to backfill if this happens again, too.

Comment 9

[Marcia Knous [:marcia]]

I see that the trunk is showing no ADUs today: https://crash-stats.mozilla.com/daily?p=Firefox&v[]=10.0a1. Laura: Should I just be filing bugs when this happens?

Comment 10

[Laura Thomson :laura]

Marcia, I'll file an IT bug for mpressman to run the backfill.

If this is continuously going to happen we're going to have to find a better way though; reopening for that.

Comment 11

[Laura Thomson :laura]

Filed https://bugzilla.mozilla.org/show_bug.cgi?id=691039 for backfill

Comment 12

[Laura Thomson :laura]

Talked to deinspanjer about running the ADU job later.  Turns out that the problem has already been solved by parallelization (before comment #9): that finishes at ~9.30pm PT and our ADU cron runs at 2am PT.

The problem in comment #9 was caused by a bug in Metrics ADU processing that affected only 10.0a1.  The problem was a regex that only recognized single digit major version numbers and has since been rectified so we should not see this again.