1300457 - Firefox Crash Data stopped updating Aug 31

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Mats Palmgren (inactive)]

The graphs stop at Aug 31:
https://crash-stats.mozilla.com/home/product/Firefox

Comment 1

[Lonnen :lonnen]

Crashes are still processing, available in super search. It looks like reports reliant on ADI #s have been failing. Investigating now...

Comment 2

[Lonnen :lonnen]

Confirmed. Our fetch-adi-from-hive job has failed 556 in the last 5 days. Once that successfully runs, everything should update into place. Now, let me figure out why that isn't working.

Comment 3

[Lonnen :lonnen]

Attachment: error from production

I've attached the error from crontabber. Permission denied at runtime. Maybe the credentials were rotated? Needs investigation.

Comment 4

[Lonnen :lonnen]

These things seem to happen every time the whole team is on an airplane. I'll have to pick this up again when I get to the airport.

Comment 5

[David Bolter [:davidb] (NeedInfo me for attention)]

[Tracking Requested - why for this release]: I don't like the idea of shipping without knowledge of crash data.

Comment 6

[Liz Henry (:lizzard) (relman/hg->git project)]

We have the crashes. Just not the idea of the crash rate by ADI. It also gives us nice information on the populations on various channels/releases so we know how updates are going, fairly quickly. 

You can see the crashes coming in, for the RC1, for example, https://crash-stats.mozilla.com/topcrashers/?product=Firefox&version=49.0b99

I am betting we will be able to backfill the ADI info as soon as the whole team isn't traveling. They've done it before!

Comment 7

[Lonnen :lonnen]

We will be able to backfill after we get ADI. The Hive cluster we rely on for ADI was upgraded, which appears to have caused a change in permissions that breaks out scripts. Pythian and the IT data team have been working on a fix.

See bug https://bugzilla.mozilla.org/show_bug.cgi?id=1300341

Comment 8

[Peter Bengtsson [:peterbe]]

The Hive server has been fixed now. https://crash-stats.mozilla.com/crashes-per-day/?p=Firefox works again.