509677 - AMO statistics have not been updated for 2 days

Status: VERIFIED FIXED

(Mozilla Metrics :: Data/Backend Reports, defect)

Description

[Norman]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)

The last add-on stats are for Saturday 8th August 2009.
They were not updated (as they usually are) on Monday 10th or on Tuesday 11th.
Millions of people download add-on's every day and the stats they see are wrong.

Reproducible: Always

Steps to Reproduce:
1. Go to Firefox add-on site
2. Look at the stats for any of the 8000+ add-on's
3. They are wrong
Actual Results:  
Look at stats for any Firefox add-on

Expected Results:  
The correct stats (sorry if this sounds stupid, but I am just filling in these boxes)

Comment 1

[Daniel Einspanjer [:dre] [:deinspanjer]]

The data processing that supports the AMO stats is currently blocked on access to the log file storage system that is undergoing repairs due to the data center outage on Sunday.
I am working with IT to attempt to determine a way around this problem since we are unsure of when the drive array will be back online.

Comment 2

[Norman]

It seems that the "weekly downloads" statistics, as shown on the https://addons.mozilla.org pages for every add-on listed are decreasing with each day the stats are not updated. 

e.g. In 6 days even major add-ons like 'Adblock Plus' will show "0 weekly downloads" on the publicly viewable web-site. This does not reflect well on Mozilla.

Comment 3

[Daniel Einspanjer [:dre] [:deinspanjer]]

Access to the log data has been restored and I've kicked off back processing for the two days of missing stats.  Once those are caught up, I'll turn on the hourly processing and then catch up the remainder of today.  It will take about 6 hours to process the two days of traffic (approximately 800 GB of uncompressed data).  If I'm still awake when it finishes, I'll push the data out to the AMO db. 

Unfortunately, we must doff our hats and mourn the passing of the hours 2009-08-09-18 and 2009-08-09-19 PDT.  IT will take another look just to make sure, but it appears that those two hours succumbed to the data center outage and we will likely be permanently missing that data (if there was even any traffic coming in to the AMO site at that time. I don't know). 

I'll post another update when back-processing is completed.

Comment 4

[Yongqian]

It's been a day, what is the status? Shouldn't back-processing have been completed by now?

Comment 5

[Daniel Einspanjer [:dre] [:deinspanjer]]

It should have been, but there has been some up and downs of the servers as IT has been working to restore the log storage drive arrays.

I have finished processing the 9th and 10th and will be uploading them shortly.
I don't think we are going to be fully caught up tonight though.

Comment 6

[Daniel Einspanjer [:dre] [:deinspanjer]]

The 9th and 10th were processed and exported to the AMO DB.  Working on the 11th and 12th right now.

Comment 7

[Daniel Einspanjer [:dre] [:deinspanjer]]

The 11th is imported.  I don't think the 12th will be ready until tomorrow morning.

Comment 8

[Norman]

The add-on stats for the 12th and the 13th has not appeared on the add-on website

Is this just a delay due to the Mozilla IT dept generally recovering from last Sunday's data center outage?
Or has there been further problems at the data center?

Comment 9

[Daniel Einspanjer [:dre] [:deinspanjer]]

12th and 13th were pushed to AMO DB today.  This should be currently caught up.

There might be another outage in the near future due to a full backup IT must make of the log file storage server, but that will be a new issue.

Comment 10

[Stephen Donner [:stephend] Not actively reading bugmail]

Norman, is this now fixed for you?

Comment 11

[Norman]

I think its fixed for everyone Stephen.
Well done Mozilla in recovering from the data outage.

Comment 12

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED -- thanks for reporting back, Norman.