422274 - Some update counts are recorded on Thursday instead of Wednesday

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Statistics, defect)

Description

[Justin Scott [:fligtar]]

spinoff from bug 421951

Need to figure out why some counts are recorded on Thursday instead of Wednesday... possibly related to timezones.

Comment 2

[Simon Paquet [:sipaq]]

Justin, any estimate on when this might be fixed? 

The statistics feature is really unusable right now (except for the total download count) as all the additional functionality (addon status, application status, OS, addon version) is based on the number of active users.

Comment 3

[Justin Scott [:fligtar]]

Gahh. I can't believe it took me this long to figure it out.

IT has a script that runs 4 different stats commands to parse the update pings. This takes several hours and is started on Thursday night, extending into Friday morning. I've been looking for problems with my script, but the problem is actually in that cron script.

We use `date --date='yesterday'` to parse for Wednesday's stats, but after it becomes Friday morning, any scripts that haven't run yet will actually be counting the logs of the new yesterday, Thursday.

oremj, can you change the script so that we set the yesterday date in a variable at the start of the script so it won't change later?

Comment 4

[Jeremy Orem [:oremj]]

Done.

#!/bin/bash

DATE=`date --date='yesterday' +%Y-%m-%d`
cd /data/stats/amo_stats/bin/parse_logs;
/usr/bin/php -f parse_logs.php logs=/data/stats/logs/im-log01/addons.mozilla.org/ geo=SJ temp=/tmp type=updatepings date=$DATE
/usr/bin/php -f parse_logs.php logs=/data/stats/logs/nladm01/addons.mozilla.org/ geo=NL temp=/tmp type=updatepings date=$DATE
/usr/bin/php -f parse_logs.php logs=/data/stats/logs/im-log01/versioncheck.addons.mozilla.org/ geo=SJ temp=/tmp type=updatepings date=$DATE/usr/bin/php -f parse_logs.php logs=/data/stats/logs/nladm01/versioncheck.addons.mozilla.org/ geo=NL temp=/tmp type=updatepings date=$DATE

Comment 5

[Justin Scott [:fligtar]]

Hmm, this line:

/usr/bin/php -f parse_logs.php
logs=/data/stats/logs/nladm01/addons.mozilla.org/ geo=NL temp=/tmp
type=updatepings date=$DATE

used to have a different date format because the logs names are like access_050908. I guess we'll need 2 variables.

Comment 6

[Jeremy Orem [:oremj]]

Good catch.

Comment 7

[Jeremy Orem [:oremj]]

SJ_DATE=`date --date='yesterday' +%Y-%m-%d`
NL_DATE=`date --date='yesterday' +%m%d%y`
cd /data/stats/amo_stats/bin/parse_logs;
/usr/bin/php -f parse_logs.php logs=/data/stats/logs/im-log01/addons.mozilla.org/ geo=SJ temp=/tmp type=updatepings date=$SJ_DATE
/usr/bin/php -f parse_logs.php logs=/data/stats/logs/nladm01/addons.mozilla.org/ geo=NL temp=/tmp type=updatepings date=$NL_DATE
/usr/bin/php -f parse_logs.php logs=/data/stats/logs/im-log01/versioncheck.addons.mozilla.org/ geo=SJ temp=/tmp type=updatepings date=$SJ_DATE
/usr/bin/php -f parse_logs.php logs=/data/stats/logs/nladm01/versioncheck.addons.mozilla.org/ geo=NL temp=/tmp type=updatepings date=$NL_DATE

Comment 8

[Justin Scott [:fligtar]]

versioncheck.addons.mozilla.org NL logs actually use the same format at SJ.

addons.mozilla.org NL is the only one that's different.

Comment 9

[Jeremy Orem [:oremj]]

SJ_DATE=`date --date='yesterday' +%Y-%m-%d`
NL_DATE=`date --date='yesterday' +%m%d%y`
cd /data/stats/amo_stats/bin/parse_logs;
/usr/bin/php -f parse_logs.php logs=/data/stats/logs/im-log01/addons.mozilla.org/ geo=SJ temp=/tmp type=updatepings date=$SJ_DATE
/usr/bin/php -f parse_logs.php logs=/data/stats/logs/nladm01/addons.mozilla.org/ geo=NL temp=/tmp type=updatepings date=$NL_DATE
/usr/bin/php -f parse_logs.php logs=/data/stats/logs/im-log01/versioncheck.addons.mozilla.org/ geo=SJ temp=/tmp type=updatepings date=$SJ_DATE
/usr/bin/php -f parse_logs.php logs=/data/stats/logs/nladm01/versioncheck.addons.mozilla.org/ geo=NL temp=/tmp type=updatepings date=$SJ_DATE