726569 - Average Daily Users count still not being updated after 8 days

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Statistics, defect)

Description

[Friedrich Haussmann]

User Agent: Mozilla/5.0 (X11; Linux i686; rv:10.0.1) Gecko/20100101 Firefox/10.0.1
Build ID: 20120210202211

Steps to reproduce:

Eight days ago the 'Average Daily Users' got set to '120'. I confirmed this number using the statistics dashboard back then ( https://addons.mozilla.org/en-US/firefox/addon/linkcheck/statistics/?last=7 ).

Yesterday I expected AMO to update the 'Average Daily Users' count, but nothing happened.



Expected results:

'Average Daily Users' count should have updated to approximately '280' according to https://addons.mozilla.org/en-US/firefox/addon/linkcheck/statistics/?last=7

I expected the 'Average Daily Users' to be updated at least once a week.

Comment 1

[support]

Same issue here. https://addons.mozilla.org/en-US/firefox/addon/donottrackplus/ Shows ~500 users but according to statistics page we have had over 15K downloads in the last couple of days.  Obviously something is out of sync.

Comment 2

[Wil Clouser [:clouserw]]

Regarding the original bug and linkcheck.  Your calculated average daily users for the last 7 days is 311.5 and 301.6 for one day before that.  The jobs appear to be running normally so I'm confirming the bug.

donottrackplus shows 4355 users for me which is correct.

Comment 3

[Friedrich Haussmann]

I investigated a little further and found with the help of the publicly available statistics many more add-ons that are affected. Most of them seem to concentrate around the 05.02.2012 with their last update of the 'Average Daily Users'. Two of them did not get updated in over a year, though the date-concentration suggests that to be a different bug.

Listed by date
'7d' number : 'Average Daily Users' for the last 7 days
'reported' number : 'Average Daily Users'

12.02.2012
7d: 9.213, reported: 8.502 - https://addons.mozilla.org/en-US/firefox/addon/abpcustomization/statistics/?last=7

11.02.2012
7d: 652, reported: 514 - https://addons.mozilla.org/en-US/firefox/addon/google-voice/statistics/?last=7

10.02.2012
7d: 743, reported: 513 - https://addons.mozilla.org/en-US/firefox/addon/noverflow/statistics/?last=7

08.02.2012
7d: 1.237, reported: 897 - https://addons.mozilla.org/en-US/firefox/addon/user-agent-quick-switch/statistics/?last=7

07.02.2012
7d: 23.999, reported: 27.174 - https://addons.mozilla.org/en-US/firefox/addon/help-menu-update/statistics/?last=7

05.02.2012
7d: 38.219, reported: 27.958 - https://addons.mozilla.org/en-US/firefox/addon/quick-maps/statistics/?last=7
7d: 24.087, reported: 24.815 - https://addons.mozilla.org/en-US/firefox/addon/google-search-link-fix/statistics/?last=7
7d: 14.783, reported: 3.719 - https://addons.mozilla.org/en-US/firefox/addon/noiascrollbars/statistics/?last=7
7d: 5.282, reported: 1.026 - https://addons.mozilla.org/en-US/firefox/addon/wwwtran/statistics/?last=7
7d: 3.240, reported: 1.788 - https://addons.mozilla.org/en-US/firefox/addon/noiabuttons/statistics/?last=7
7d: 1.764, reported: 1.401 - https://addons.mozilla.org/en-US/firefox/addon/cheevos/statistics/?last=7
7d: 881, reported: 676 - https://addons.mozilla.org/en-US/firefox/addon/mind-the-time/statistics/?last=7
7d: 614, reported: 541 - https://addons.mozilla.org/en-US/firefox/addon/classic-rsgb/statistics/?last=7

365+ days
7d: 3.521.971, reported: 321.535 - https://addons.mozilla.org/en-US/firefox/addon/test-pilot/statistics/?last=365
7d: 92.306, reported: 19.972 - https://addons.mozilla.org/en-US/firefox/addon/url-fixer-plus-ru-and-ua/statistics/?last=365

Comment 4

[Wil Clouser [:clouserw]]

I added some debug logging in https://github.com/mozilla/zamboni/commit/e6b2a2839cd93358a393e2b11a0fce736cc444a4

Comment 5

[Wil Clouser [:clouserw]]

I think this is being caused by inconsistent celery tasks.  Following are the totals for add-ons calculated since I added logging.  They should be all very close to the same, only increasing by the number of add-ons added per day.  I think the right number is closer to 15k.

2012-03-04 10915
2012-03-03 8181
2012-03-02 11055
2012-03-01 10933
2012-02-29 7893
2012-02-28 9424
2012-02-27 11964
2012-02-26 10058
2012-02-25 10114
2012-02-24 11286

Comment 6

[Wil Clouser [:clouserw]]

Actually, I have the totals sent too.  First column is date, second is number sent to celery *as reported by the celery task*, 3rd is the number updated (2 numbers for both sides of the if statement in the code):

2012-03-04 15525 10915+56
2012-03-03 15531 8181+40
2012-03-02 15521 11055+51
2012-03-01 15515 10933+51
2012-02-29 15522 7893+29
2012-02-28 15494 9424+47
2012-02-27 15462 11964+63
2012-02-26 15460 10058+47
2012-02-25 15462 10114+49
2012-02-24 15454 11286+54

Comment 7

[Wil Clouser [:clouserw]]

Looks like we don't get exception reports from celery tasks and this was failing because some add-ons don't exist.  I dropped in a patch to fix this up and in our next push we should be good to go.  Thanks for the report and thanks to oremj for investigating.

https://github.com/mozilla/zamboni/commit/08c891da28569acbb609f6bd9ce98d7614822ac2