683439 - Active daily user count dropped significantly on 2011-08-30

Status: RESOLVED WONTFIX

(addons.mozilla.org Graveyard :: Statistics, defect)

Description

[testit]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b13pre) Gecko/20110303 Firefox/4.0b13pre
Build ID: 20110303122446

Steps to reproduce:

The active daily user count for 30th of August dropped significantly. Please check e.g. here:
https://addons.mozilla.org/en-US/statistics/addon/1865
https://addons.mozilla.org/en-US/statistics/addon/433

Comment 1

[Wladimir Palant]

Confirming.

Comment 2

[Daniel Einspanjer [:dre] [:deinspanjer]]

Hey guys, not sure what is going on here.  I'd normally suspect the migration of AMO to phx datacenter, but I can't think of any way that would not also affect the 28th and 29th as well.

I will investigate tonight and first thing tomorrow and report back by morning Pacific.

Comment 3

[testit]

The same happened on 31th.

Comment 4

[Meredith]

For Easy YouTube Video Downloader we see a 94.91% drop for yesterday's data - August 31st.  Is there an update on the cause of this issue or time to resolution?

Comment 5

[Daniel Einspanjer [:dre] [:deinspanjer]]

IT made some sort of change on the 30th. I'm trying to get details to figure out whether that might have been the cause.

Comment 6

[Daniel Einspanjer [:dre] [:deinspanjer]]

Okay.  The changes that were pushed on the 30th caused us to be missing some data because it arrived after the max time we wait to process logs for each hour.

I configured the process to be forced to wait for data from all the new servers, and it is picking up the missing data from today.  That said, IT has reported having some intermittent problems with our web serving clusters today and there might still be a *slight* dip.

After we get today caught up, I'll work on setting up a backfill process to take care of the previous two days.

Comment 7

[Daniel Einspanjer [:dre] [:deinspanjer]]

Getting caught up with all the missing data from today has been very slow going.  We are still about 4 hours behind, and we have now passed the normally scheduled time for our End of Day (EOD) processes to start so metrics exports will be delayed.

Comment 8

[Daniel Einspanjer [:dre] [:deinspanjer]]

Okay, The data for yesterday looks more reasonable.  That said, please understand that AMO suffered a 3 hour outage on the morning of the 1st UTC, and as such, we have a 10% to 15% depression in the ADU numbers across the board.

I will try to set up a back processing script to import data for the previous two days and run it over the weekend.

Comment 9

[testit]

The data from 30/31 seems to be still unchanged.

Comment 10

[Daniel Einspanjer [:dre] [:deinspanjer]]

I had to abort the back-processing due to disk space issues.  We are working on clearing up some room so we can resume this, but it is likely to take a while. I'll give an update at the end of the week.

Comment 11

[Dave Garrett]

Did you manage to resolve the issue preventing the back-processing?

Comment 12

[Daniel Einspanjer [:dre] [:deinspanjer]]

Ugh, sorry I didn't post an update on this thread.  We have not yet resolved the issue with lack of resources on the current cluster.  We have a new replacement cluster that is being built out, but it had to be reformatted because the wrong distro was installed. :/
While I would be happy to back-process the data if we can easily get to it when the new cluster is up, I would like to know how critical filling in this missing data is.  Is it something we could live without since we have the correct data flowing in at this point?

Comment 13

[Dave Garrett]

I would consider this low priority unless anyone else says otherwise. It would still be helpful to get it fixed eventually as the hole in the data messes up things a little bit when my stat checking scripts graph a few things. Certainly no rush though.

Comment 15

[Wil Clouser [:clouserw]]

Daniel: any news on this?

Comment 16

[Daniel Einspanjer [:dre] [:deinspanjer]]

We have the new cluster online, and it has made it possible for us to do the most recent reprocessing that was needed for 2011-12-15.  We don't have the manpower necessary to try to dig up older data to backfill holes at this point though, so I would prefer to close this bug out.

Bug 708539 was the more recent outage where we backfilled for 12-15.

Comment 17

[Simon Paquet [:sipaq]]

Marking WONTFIX per comment 16.