Closed Bug 1089358 Opened 10 years ago Closed 10 years ago

Statistics are missing from 2014-10-21 through 2014-10-28, inclusive

Categories

(addons.mozilla.org Graveyard :: Statistics, defect, P2)

Product:
addons.mozilla.org Graveyard
Component:
Statistics
Type:
defect

Tracking

(Not tracked)

Status:
VERIFIED FIXED
Milestone:
2014-12

People

(Reporter: frfxtst, Assigned: jlaz)

References

Details

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0
Build ID: 20141011015303

Steps to reproduce:

The daily users and download numbers are missing in the statistics since 2014-10-20. Check e.g. here https://addons.mozilla.org/en-US/firefox/addon/adblock-plus/statistics/?last=30.
Same as bug 1088639
No, to the same thing - bug 1088639 is about overall AMO statistics, this bug is about add-on statistics however. I can confirm that the stats are gone, nothing is being listed any more since 2014-10-20 and that apparently affects all extensions.

Jorge, do you know how is responsible for this?
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(jorge)
Red Rover, Red Rover, I call Will Clouser over.
Flags: needinfo?(clouserw)
Not sure what broke.  The current system is no longer supported so I don't know if we'll be able to fix it.  You'll probably be interested in watching bug 1059966, particularly bug 1070238.
Flags: needinfo?(clouserw)
In reply to Wil Clouser [:clouserw] from comment #5)
> Not sure what broke.  The current system is no longer supported so I don't
> know if we'll be able to fix it.  You'll probably be interested in watching
> bug 1059966, particularly bug 1070238.

So the old system no longer works but the new system is not live?
Any news? Today is the ninth day without new statistics.
Wil, is the new system expected to be pushed on mid-November? Could it be uplifted?
Flags: needinfo?(jorge) → needinfo?(clouserw)
Hoping to push it today.  Watch #amo for news.
Flags: needinfo?(clouserw)
Summary: Daily users and download statics not updated since 2014-10-20 → Statistics are missing from 2014-10-21 through 2014-10-28, inclusive
So the new system is now live, and we need to backfill the data, in four steps:

1/ (per day) query hive for the download and update counts for addons, store them in files on the peach servers, and scp them to the amo server
2/ (per day) fill in the database on amo, from the data stored in files
3/ (per day) query hive for the theme update counts, store the data in files locally, and fill in the database on amo from those files
4/ (once) compute the theme popularity and movers



** First, log in to the peach server, using the amo_prod user, and run the following commands, for each day (so for "2014-10-21", "2014-10-22", ... "2014-10-28"):

- /usr/bin/python /home/amo_prod/amo_download_counts/amoDownloadCountsFiltered.py #date# >> /home/amo_prod/logs/amoDownloadCountsFiltered.txt 2>&1
- /usr/bin/python /home/amo_prod/amo_update_counts/amoUpdateCountsFiltered.py #date# >> /home/amo_prod/logs/amoUpdateCountsFiltered.txt 2>&1

This will take a few (10?) minutes per "day" for the amoDownloadCountsFiltered, but it'll take much longer for the amoUpdateCountsFiltered (2 hours?).

Once that is all done, all the files should be scp'ed on the amo production servers.

** Second, log on the amo production server, and run the following management commands for each day:

- DJANGO_SETTINGS_MODULE=settings_local python manage.py download_counts_from_file --date #date#
- DJANGO_SETTINGS_MODULE=settings_local python manage.py update_counts_from_file --date #date#

Those two should run in a few seconds, or minutes at most.

** Third, still on the amo production server, run the following management commands for each day:

- DJANGO_SETTINGS_MODULE=settings_local python manage.py theme_update_counts_from_hive --date #date#
- DJANGO_SETTINGS_MODULE=settings_local python manage.py theme_update_counts_from_file --date #date#

Those two should run in a few minutes.

** Fourth, still on the amo production server, when all the rest is done, run the following management command once:

- DJANGO_SETTINGS_MODULE=settings_local python manage.py update_theme_popularity_movers

This should run in a few minutes.



To spare some disk space on amo, you can run the "*_from_file" from step 2 as soon as the corresponding day has been generated on peach in step 1. At the end of each "*_from_file" command, the file on disk should be automatically removed on amo.
Assignee: nobody → jlaz
It seems you also need to run the following command, after step 4:

- DJANGO_SETTINGS_MODULE=settings_local python manage.py index_stats --date=2014-10-21:2014-10-28
Backfill commands in comment 10 and comment 11 completed. Please verify.
doesn't show up for me yet, but let's check again in the morning in case of caching
All fine here now, worked correctly from what I can tell.
Looks good to me now, for the per add-on statistics. The global statistics at https://addons.mozilla.org/en-US/statistics/addons_in_use/?last=30 still show up blank.
The issue from comment 15 is unrelated I believe, and there's a bug for it: https://bugzilla.mozilla.org/show_bug.cgi?id=1088639
Right, I thought that bug was fixed. My bad...

What does not work yet, is when you group by week. That data is still missing from the stats. I suppose there is maybe another index that needs to be updated for that?
Excellent, this is looking good this morning for me too.  Group by week also works for me - make sure you're showing several weeks worth of data for it to work.
On one of my computers the data for october 20 .. 28 keeps missing, but I suppose this issue is fixed and can be closed
The data retrieved from the servers are stored locally (in the localstorage), and never retrieved again after that. So if you had seen those before, they won't be updated anymore.

The "fix" is to remove the localstorage in your javascript console using "localStorage.clear()".
Language statistics are still down for languages including the 2-letter region code (e.g. en-US, en-GB, pt-BR, etc.) in contrast to languages without region code (e.g. de, fr, etc.)

This means the data is not only missing from october 21st to 28th but also no new data is generated.
(In reply to Eduard Braun from comment #21)
> Language statistics are still down for languages including the 2-letter
> region code (e.g. en-US, en-GB, pt-BR, etc.) in contrast to languages
> without region code (e.g. de, fr, etc.)

While I can confirm that issue, this is definitely a different issue related to the new HIVE metrics system (bug 1059966). It needs to be filed as a separate bug.
I filed bug 1095113 for the issue with locales including a region-code.
Depends on: 1093699
We need to re-run the backfilling commands once bug 1093699 is pushed to prod. There was a bug related to the exclusion of all non-addons stuff.
Priority: -- → P2
Target Milestone: --- → 2014-11
After speaking to magopian on IRC, we will begin work on this after the proposed prod push on Thursday, 11/20, to backfill stats from 2014-10-21 up until the day of the push
Any news? Oct 21 - Nov 20 stats are completely absent.
Flags: needinfo?(mathieu)
Flags: needinfo?(jlaz)
Backfill is currently downloading stats up to the date of 11/15, we should be done by Friday at this rate
Flags: needinfo?(jlaz)
Flags: needinfo?(mathieu)
Target Milestone: 2014-11 → 2014-12
Stats from 10/21-11/20 should now be loaded.  Please refer to comment 20 to ensure that you are looking at the most current state for global statistics.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Verified as fixed . All stats are now displayed
Status: RESOLVED → VERIFIED
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.