920313 - Discrepancy between Google Analytics reports and Stats reported to Developers via Dev Hub

Status: RESOLVED FIXED

(Marketplace Graveyard :: Statistics, defect, P3)

Description

[Karen Ward [:kward]]

Attachment: Copy of age of barbarians installs last 365 days.xls from Dev Hub

App install count is inconsistent between Developer Hub, Marketplace Extract and Google Analytics Reports.
Example: Dev Hub: Age of Barbarians shows a total of 253 installs since publication 365 days; 32 last week.  Google Analytics show a total of 1162 installs since publication.
Example: Marketplace Extract that matches Dev hub counts show 5405 installs last week, Google Analytics report shows 34,591 installs last week.

We need to know what is accurate for Marketplace, to share with Operators and also to share with Developers for their own apps so I am marking this as a Major Bug.

See Attachments.

Comment 1

[Karen Ward [:kward]]

Attachment: Analytics marketplace.firefox.com (FxOS only) Firefox OS Marketplace Overview 20130915-20130921.pdf

Total downloads of 34591 from Google analytics for week of 9/15-9/21

Comment 2

[Karen Ward [:kward]]

Attachment: 2013 09 20 Listed and Queued Apps.xlsx

Total downloads of apps week ending 9/20. Count Matches Dev Hub - 5405 total.

Comment 3

[Wil Clouser [:clouserw]]

Google Analytics is the source of truth for app installs

Comment 4

[Matt Basta [:basta]]

This discrepancy happens as a result of the way we count installations. The major reason for the difference is that we don't count anonymous installs in zamboni. Other variables contribute, but this is the primary reason.

For the foreseeable future, this will continue to be the case. It's unclear whether event data can be extracted from GA, so we should be providing the carriers with the reports from Gareth Cull and Chris More, or creating new GA reports when they are requested.

Comment 5

[Andrew Williamson [:eviljeff]]

... as install is the only stat we show developers and its wrong, shouldn't we disable /app/<slug>/statistics entirely for now?  (Or fix it?)

Comment 6

[Rob Hudson [:robhudson]]

(In reply to Matt Basta [:basta] from comment #4)
> This discrepancy happens as a result of the way we count installations. The
> major reason for the difference is that we don't count anonymous installs in
> zamboni. Other variables contribute, but this is the primary reason.

For per-app install counts this is still true unfortunately, but it can be fixed. We fixed this for global install counts recently. For global installs we count both anonymous and logged-in user installs as long as the front-end completes the request to the install API in zamboni. Both record to the monolith table in code[1] and eventually get into Monolith. And until we fully switch to Monolith based stats I'm extracting the counts out of this table for the install counts for the current stats[2].

[1] https://github.com/mozilla/zamboni/blob/master/mkt/installs/api.py#L39
[2] https://github.com/mozilla/zamboni/blob/master/apps/stats/tasks.py#L242-L243

> For the foreseeable future, this will continue to be the case. It's unclear
> whether event data can be extracted from GA, so we should be providing the
> carriers with the reports from Gareth Cull and Chris More, or creating new
> GA reports when they are requested.

I could do the same for per-app installs which currently pulls from the `Installed` table which only counts logged in users[3]. The switch would be similar to [2] and pull from the monolith table temporarily until Monolith officially takes this over, at which point the code at [3] will be removed.

[3] https://github.com/mozilla/zamboni/blob/master/mkt/stats/tasks.py#L339

Depending on how close the Monolith stats are this may or may not be worth it.

Comment 7

[Matt Basta [:basta]]

If it's a relatively simple change to start counting anonymous installs for monolith, then I think we should at least do that and see what the numbers look like.

Comment 8

[Andrew Williamson [:eviljeff]]

re: anonymous installs - https://bugzilla.mozilla.org/show_bug.cgi?id=836586 was supposed to address this and was marked fixed back in March.  Did something regress?

Comment 9

[Rob Hudson [:robhudson]]

(In reply to Andrew Williamson [:eviljeff] from comment #8)
> re: anonymous installs - https://bugzilla.mozilla.org/show_bug.cgi?id=836586
> was supposed to address this and was marked fixed back in March.  Did
> something regress?

The links in bug 836586 don't address this. I attempted to fix this in bug 910824 but failed to notice that there were two places in the code that aggregate these and only got the one.

Comment 10

[Thomas Elin [:relaas]]

Just commenting here as this is still presenting an issue for our developers.

Today, Zeptolabs (Owners of Cut the Rope) were concerned that there app has only received 500 installs since it went live a few days ago. Checking google analytics it turns our the number is closer to 3000.

It's super important that we get this fixed asap. Continually under reporting app downloads will only discourage developers from putting more time and resources into their apps and the marketplace in general.

Comment 11

[Karen Ward [:kward]]

(In reply to thomas elin from comment #10)
> Just commenting here as this is still presenting an issue for our developers.
> 
> Today, Zeptolabs (Owners of Cut the Rope) were concerned that there app has
> only received 500 installs since it went live a few days ago. Checking
> google analytics it turns our the number is closer to 3000.
> 
> It's super important that we get this fixed asap. Continually under
> reporting app downloads will only discourage developers from putting more
> time and resources into their apps and the marketplace in general.

+1 from me.  We need these numbers to be accurate for both the developers and the Carriers.  Right now we are presenting conflicting data.

Comment 12

[Wil Clouser [:clouserw]]

This is already on track to fix for next week.

Comment 13

[Thomas Elin [:relaas]]

Can I get a update on this bug? Are we still planning to push this fix into production today?

Comment 14

[Wil Clouser [:clouserw]]

We do pushes on Tuesdays, so no, but it sounds like we still need to investigate more.  Bug 933406 may be a duplicate of this one, but it has some more info in it.  These may be caused by DNT flags or might be more bugs.

At this point in time, we've disabled the devhub dashboards while we figure out the numbers.  If people have questions, give them the google numbers.

Comment 15

[Rob Hudson [:robhudson]]

We switched to using GA as the source for install counts in bug 938897. I'm not sure there's anything to do in this bug.

Comment 16

[Thomas Elin [:relaas]]

I am unable to verify this fix has been implemented as the Statistics page within Marketplace is still down and displaying a notification, "Statistics pages are under construction. Check back soon.".

The primary concern for this bug was that inaccurate data was being presented to our app developers. Reopened as we are not currently presenting download metrics via the Marketplace.

Comment 17

[Matt Basta [:basta]]

If we're not presenting any data, then we're not presenting incorrect data, yeah? See no stats, hear no stats, serve no stats

Comment 18

[Rob Hudson [:robhudson]]

(In reply to thomas elin from comment #16)
> I am unable to verify this fix has been implemented as the Statistics page
> within Marketplace is still down and displaying a notification, "Statistics
> pages are under construction. Check back soon.".
> 
> The primary concern for this bug was that inaccurate data was being
> presented to our app developers. Reopened as we are not currently presenting
> download metrics via the Marketplace.

What method did you use to find the page with the under construction message? Stats pages exist but we haven't updated the links yet b/c we want to verify that the data is accurate first.

See:
https://marketplace-dev.allizom.org/stats/ for global stats and
https://marketplace-dev.allizom.org/stats/app/twitter/ for per-app stats.

Comment 19

[Thomas Elin [:relaas]]

I was checking production Marketplace. Do we know when this fix will go live?

https://marketplace.firefox.com/app/cut-the-rope/statistics/

Comment 20

[Rob Hudson [:robhudson]]

As soon as it is verified we will update the URL to the stats page to the new one. The old URL will be deprecated. So wherever you found that URL we will update it to point to https://marketplace.firefox.com/stats/app/cut-the-rope/

Comment 21

[Thomas Elin [:relaas]]

Taking a look at the new statistics page and comparing that data against Google Analytics it looks identical.

I've only checked a total of 15 days for the Cut the Rope app but again the #'s between Marketplace and GA were exactly the same.

Let's try to get this fix verified asap and the new statistics page live on the Marketplace. We have a number of top tier developers who are waiting to see how their apps are performing.