880980 - addon stats graph missing data (probably ADU) for many days

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Public Pages, defect)

Description

[Selena Deckelmann :selenamarie :selena]

Original report here https://bugzilla.mozilla.org/show_bug.cgi?id=876143#c19


This issue was attached to an unrelated issue regarding ADU pushes into Socorro. I'm hoping this is the right place to send this bug!

On: 
https://addons.mozilla.org/en-US/firefox/addon/firefox-2-theme-for-firefox-3x/statistics/?last=90

Stats missing/zero:
April 29/30, May 1-30 (1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30)

Extensions:
AdblockPlus: [May 2, 7-8, 16-19, 21, 25]
DownThemAll: [May 2, 4, 6, 14-15, 21, 25, 27-28, 30]
GreaseMonkey: [Apr 29-30] [May 1, 7, 10, 12, 19]
Firebug [Apr 30] [May 5, 10, 12, 19, 22, 25]
FireFTP [May 2, 12, 14, 19, 29]
Stylish [May 16-19, 21, 23, 25, 30]

Themes:
A Blue Fox Theme: [Apr: 30; May: 4-5, 14-17, 20, 26]
Bloody Red theme pack - 5 themes in one [May 4, 11, 14-15, 17, 26]
Classic Compact: [May: 3, 9, 17, 21, 24, 30]
Firefox 2, the theme, reloaded [Apr 29, May 1, 7, 17, 20, 22]
FXChrome [Apr 29] [May 12, 18-20]
Noia 4 [Apr 29, May 4, 18-20, 22, 26]
Silvermel: [May 5, 7, 18-19, 21, 28]

Comment 1

[Daniel Einspanjer [:dre] [:deinspanjer]]

I did some investigation on the AMO read only slave and for the all add-ons aggregate as well as Adblock Plus, there are ADI for each of these days in the update_counts table.  Something else seems to be going on.

Comment 2

[Daniel Einspanjer [:dre] [:deinspanjer]]

Here are the queries I ran on the AMO read only DB slave:
select date, count(*), sum(count) from update_counts where date > '2013-05-01' group by 1;
select date, count(*), sum(count) from update_counts where date > '2013-05-01' and addon_id = 1865 group by 1;

Comment 3

[Selena Deckelmann :selenamarie :selena]

(In reply to Daniel Einspanjer :dre [:deinspanjer] from comment #2)
> Here are the queries I ran on the AMO read only DB slave:
> select date, count(*), sum(count) from update_counts where date >
> '2013-05-01' group by 1;
> select date, count(*), sum(count) from update_counts where date >
> '2013-05-01' and addon_id = 1865 group by 1;

Thanks, Daniel.  I feel at a loss - I have no access to these systems, nor do I know who the right person to contact next to do something about these issues.

Comment 4

[Daniel Einspanjer [:dre] [:deinspanjer]]

I agree.  Since it isn't on the DB side, I don't personally know who maintains the website side of things anymore.  But I think the bug is in the right place, hopefully someone will pick it up and respond today.

Comment 5

[Wil Clouser [:clouserw]]

We reindexed the stats (which takes ~30 hours or so) and this fixed most of the holes

Eg, fixed:
> https://addons.mozilla.org/en-US/firefox/addon/adblock-plus/statistics/?last=90
> https://addons.mozilla.org/en-US/firefox/addon/downthemall/statistics/?last=90

Still has a couple of holes:
> https://addons.mozilla.org/en-US/firefox/addon/firefox-2-theme-for-firefox-3x/statistics/?last=90
> https://addons.mozilla.org/en-US/firefox/addon/greasemonkey/statistics/?last=90
> https://addons.mozilla.org/en-US/firefox/addon/firebug/statistics/?last=90
> https://addons.mozilla.org/en-US/firefox/addon/fireftp/statistics/?last=90
> https://addons.mozilla.org/en-US/firefox/addon/stylish/statistics/?last=90


The data is in the db for the ones with holes still, but isn't in the indexes.  It did 900k+ jobs this weekend, so maybe some got dropped.

Comment 6

[Wil Clouser [:clouserw]]

Ran `fixup` here.  We're good now.

Comment 7

[Mark Capella [:capella]]

Wait a sec... maybe the data got caught up through the 12th, but I don't see it staying caught up ... ?

https://addons.mozilla.org/en-US/android/addon/quitnow/statistics/?last=90
https://addons.mozilla.org/en-US/firefox/addon/greasemonkey/statistics/?last=90

Is there an issue with your upstream feed?

Comment 8

[Wil Clouser [:clouserw]]

The issue is actually that the offline processing isn't happening in a timely manner.  There were 900000 jobs in the queue as of 8pm last night.  I'm not sure where it's at today.  There are multiple bugs talking about it now though, so let's just keep everything in bug 882659.  Thanks!

Comment 9

[Mark Capella [:capella]]

Great! Didn't see a cross ref to that bug# until you just posted it! Thanks :D