904747 - Mysterious missing stats

Status: RESOLVED WONTFIX

(addons.mozilla.org Graveyard :: Statistics, defect, P2)

Description

[Wil Clouser [:clouserw]]

We've been missing stats periodically for weeks (months?).  The fixup job seems to fill in the holes so the data is there, it's just the daily calculations which fail occasionally (dropped/failing celery jobs?).  They are scattered all over the site but https://addons.mozilla.org/en-US/firefox/addon/adblock-plus/statistics/?last=30 is an example

Comment 1

[Anurag Phadke[:[e-mail]]]

IT moved from Zeus to Netscaler on 7/24, netscaler had config issue which resulted in barely any logging for 7/24.

On 8/8, there was partial failure on Netscaler which resulted in missing logs. Metrics team (mainly Daniel) is working on recovering some of the log data for 8/8 and we should have updated counts for 8/8 by EOD.

-anurag

Comment 2

[Wil Clouser [:clouserw]]

Thanks Anurag.  This has been happening for a long time - we have mysterious dropped stats and we've been running fixup for a while.  Would be nice to fix it for good.

Comment 3

[Anurag Phadke[:[e-mail]]]

Wil - can u please send me the dates (or bugs) where we have had dropped stats? I'll look in it and see how we can ensure this to not happen again.

thx,
-anurag

Comment 4

[Anurag Phadke[:[e-mail]]]

Based on https://addons.mozilla.org/en-US/firefox/addon/adblock-plus/statistics/?last=365, it seems that we have had issues on 6/21, 7/24 and 8/8. 

We have monitoring in place that will allow us to capture the drops lot quicker than before.

Comment 5

[Wil Clouser [:clouserw]]

Anurag - I think you might be misunderstanding.  This is a marketplace bug, not yours. :)  I just CC'd you because we talked about backfilling missing stats yesterday and I just wanted to keep you in the loop.  Nothing for you to do here.  Thanks though!

Comment 6

[Anurag Phadke[:[e-mail]]]

thx wil! 
:)

Comment 7

[Wil Clouser [:clouserw]]

Thanks for filing this.  Due to resource constraints we are closing bugs which we won't realistically be able to fix.  If you have a patch that applies to this bug please reopen.

For more info see http://micropipes.com/blog/2014/09/24/the-great-add-on-bug-triage/