784157 - Google Play Campaign Tracking

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Karen Rudnitski [:kar]]

PURPOSE:
 
Implement Google Play Store Campaign Tracking (link: http://bit.ly/QNpagA) to provide insight into the success of marketing campaigns and improve download conversion rates. 


REASONING:

Product Marketing and Engagement is planning on running acquisition campaigns for Firefox for Android to drive downloads. Some of these efforts--in particular a TBD viral campaign--will involve campaigns leveraging multiple paid/owned/referred channels. At the moment, though, we aren't capable of distinguishing the effect that individual campaigns have on downloads because they overlap. Without this granularity, we won't be able to gauge the success (for example) of a spend of $X on twitter and $X on FB, and thus lack the ability to effectively divert assets to successful campaigns from floundering ones.


GOOGLE PLAY STORE CAMPAIGN TRACKING:
 
Google Play Store Campaign tracks individual marketing campaigns in aggregate, and does not reveal personally identifiable information. How it works:
1. Marketing URL links are created using the Google Play URL Builder
2. Upon the initiation of the download, a broadcastreceiver responds and indicates which campaign initiated the downloads. From developer documentation, 
“The following BroadcastReceiver allows your app to respond to the INSTALL_REFERRER intent broadcast by the Google Play Store when your app is installed. Add it to your AndroidManifest.xml file as follows: <!-- Used for install referrer tracking-->
<receiver android:name="com.google.analytics.tracking.android.AnalyticsReceiver" android:exported="true">
  <intent-filter>
    <action android:name="com.android.vending.INSTALL_REFERRER" />
  </intent-filter>
</receiver>

3. Campaign URLS are set in Google Play Store Links. From developer documentation: 
Campaign Parameters are used to pass information about the campaign or traffic source that referred a user to your app's Google Play Store page into your app's Google Analytics implementation.
To learn how to build a campaign parameter string, use the Google Play URL Builder, or consult the Campaign Parameters reference section.
Once you've built your campaign parameter string, add it to your Google Play Store URLs as the value of the referrer parameter


THE REQUEST:

Can this please be implemented? We don't believe it requires RelEng support (from an earlier email thread).

Comment 1

[Mark Finkle (:mfinkle) (use needinfo?)]

What I am wondering about is: What does Fennec do with INSTALL_REFERRER we receive? Frankly, I don't think the app cares at all how it was installed. How does the data get back to Mozilla?

Comment 2

[Allison Naaktgeboren :ally]

making sure the privacy review & the work stay linked :)

Comment 3

[Mark Finkle (:mfinkle) (use needinfo?)]

Attachment: patch

This patch adds the needed code to handle INSTALL_REFERRER broadcasts. The receiver will:
* Check for referrer extra data
* Extract the referrer data
* Make sure there is a utm_source that matches "mozilla" (ADXTEST for testing)
* Send "playstore" to Gecko preferences as the distribution.id
* Send the campaign name to Gecko preferences as the distribution.version

The distribution data should then be sent to Mozilla Metrics.

Testing this is a bit of a pain. I used a test app from the Play Store to send a broadcast. I then checked the "distribution" preferences in about:config to make sure the campaign name is in distribution.version and "playstore" is distribution.id

Android test app: https://play.google.com/store/apps/details?id=com.AdX.AdXReferralTest&hl=en

We still need to put this patch in a play store build, install via Google Tracking URL and see the distribution data in a Mozilla Metrics dashboard.

Comment 4

[Brad Lassey [:blassey] (use needinfo?)]

Comment on attachment 666873 [details] [diff] [review]
patch

Review of attachment 666873 [details] [diff] [review]:
-----------------------------------------------------------------

::: mobile/android/base/ReferrerReceiver.java
@@ +30,5 @@
> +        if (ACTION_INSTALL_REFERRER.equals(intent.getAction())) {
> +            String referrer = intent.getStringExtra("referrer");
> +            if (referrer == null)
> +                return;
> + 

ws

@@ +33,5 @@
> +                return;
> + 
> +            HashMap<String, String> values = new HashMap<String, String>();
> +            try {
> +                String referrers[] = referrer.split("&");

this is a URL right? Use Uri.getQueryParameter()

http://developer.android.com/reference/android/net/Uri.html#getQueryParameter%28java.lang.String%29

@@ +47,5 @@
> +                    JSONObject jsonPref = new JSONObject();
> +                    jsonPref.put("name", "distribution.id");
> +                    jsonPref.put("type", "string");
> +                    jsonPref.put("value", "playstore");
> +        

ws

@@ +54,5 @@
> +                } catch (JSONException e) {
> +                    Log.e(LOGTAG, "Error setting pref [distribution.id]", e);
> +                }
> +
> +                try {

if we can't set the distribution id, shouldn't we bail on setting the distribution value as well? Will that be useful at all?

@@ +59,5 @@
> +                    JSONObject jsonPref = new JSONObject();
> +                    jsonPref.put("name", "distribution.version");
> +                    jsonPref.put("type", "string");
> +                    jsonPref.put("value", values.get("utm_campaign"));
> +        

ws (I have a great emacs macro you might be interested in)

Comment 5

[Mark Finkle (:mfinkle) (use needinfo?)]

Attachment: patch v2

This patch fixes some of the comments:
* Removes any leading ws
* Re-orgs the code to try to create the JSON objects before sending the events. This reduces the chances of only sending one of the two prefs.
* I tried using the Uri class, but it seems to choke on the referrer string, which is not a real URL
* Made "mozilla" be the source. This breaks using the ADX Referral Test app to test. We'll need metrics help to verify.

Tested (before the "mozilla" source change) and it still updates the distribution prefs correctly.

Comment 6

[Mark Finkle (:mfinkle) (use needinfo?)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/e444e0facfb5

Comment 7

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/e444e0facfb5

Comment 8

[Alex Keybl [:akeybl]]

Comment on attachment 667065 [details] [diff] [review]
patch v2

[Triage Comment]
We're considering this bug for FF16 inclusion. Pre-approving for Aurora 17 to make any regressions unrelated to being downloaded from the store clear.

Comment 9

[Mark Finkle (:mfinkle) (use needinfo?)]

To test this feature:
* You must be on an Android device. Using desktop browsers and web-to-phone installs will not work [1]
* Using the stock or chrome browser, visit: http://people.mozilla.com/~mfinkle/playtrack.html
* Tap either link
* Choose to open the link in Play Store
* Install the test app (SplitReleaseApp) - It's an unofficial Fennec build
* Open the test app
* Open "about:config" and search for "distribution" prefs

You should see "playstore" and "test1" (or "test2" depends on the link you clicked). Now the blocklist ping does the rest. We need to get verification from Metrics that the data is being collected.

[1] https://developers.google.com/analytics/devguides/collection/android/v2/campaigns#known-issues

Comment 10

[Aaron Train [:aaronmt]]

Samsung Galaxy Note (Android 4.0.4)

* Test1 Campaign (via Chrome)
** distribution.id; playstore
** distribution.version; test1

* Test2 Campaign (via Chrome)
** distribution.id; playstore
** distribution.version; test2

* Test 1 Campaign (via Stock)
** distribution.id; playstore
** distribution.version; test1

* Test 2 Campaign (via Stock)
** distribution.id; playstore
** distribution.version; test2

Comment 11

[Mark Finkle (:mfinkle) (use needinfo?)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/dfb5803b7204

Comment 12

[Aaron Train [:aaronmt]]

(In reply to Mark Finkle (:mfinkle) from comment #9)
> We need to get verification from Metrics that the data is being collected.

This happening today?

Comment 13

[Mark Finkle (:mfinkle) (use needinfo?)]

I talked to Metrics. They could not find our campaign data. We debugged it a bit and I think the issue is in the way we send the distribution data to Metrics in the blocklist ping.

We only grab the _default_ version of the preference, not the _user_ set value. Since we are setting the value of the preference at runtime, it is treated as a user value.

I can change that, but the preference would only remain set during the duration of the session. It would be reset to "default" after a restart.

I could also change the blocklist code on Android to use _user_ values in the blocklist ping.

Comment 14

[Aaron Train [:aaronmt]]

Re-testing newly added APK

Samsung Galaxy Nexus (Android 4.1.1)

* Test1 Campaign (via Chrome)
** distribution.id -- non-existant preference 
** distribution.version -- non-existant preference

* Test2 Campaign (via Chrome)
** distribution.id -- non-existant preference 
** distribution.version -- non-existant preference

* Test1 Campaign (via Stock)
** distribution.id; playstore (new default)
** distribution.version; test1 (new default)

* Test2 Campaign (via Stock)
** distribution.id; playstore (new default)
** distribution.version; test2 (new default)

Comment 15

[Aaron Train [:aaronmt]]

(In reply to Aaron Train [:aaronmt] from comment #14)
> Samsung Galaxy Nexus (Android 4.1.1)
> 
> * Test1 Campaign (via Chrome)
> ** distribution.id -- non-existant preference 
> ** distribution.version -- non-existant preference
> 
> * Test2 Campaign (via Chrome)
> ** distribution.id -- non-existant preference 
> ** distribution.version -- non-existant preference

^ Google Play (Website) remote-install 
-----------------------------------

Opening with the Google Play (vending APK)

*Test1 Campaign (via Chrome)
** distribution.id; playstore
** distribution.version; test1

* Test2 Campaign (via Chrome)
** distribution.id; playstore
** distribution.version; test2

Comment 16

[Mark Finkle (:mfinkle) (use needinfo?)]

Attachment: patch 2 v1

This patch gets things working better. I am still trying to verify with Metrics that data is being delivered, but the preferences seem to be set up correctly now.

The patch:
* Uses a specific JSON message instead of treating the data like prefs
* Creates a Distribution class in browser JS to handle two things:
  * The incoming JSON message from Java
  * Resetting the distribution preferences as default prefs. This only works for the life of the application, so it needs to be done each time the app starts.

* We store the distribution data in /data/data/org.mozilla.fennec/distribution.json
* The file is saved and loaded async using NetUtils, so it should not hurt startup time.

It turns out that desktop Firefox does the same kinda thing with the DistributionCustomizer.jsm module:
http://mxr.mozilla.org/mozilla-central/source/browser/components/distribution.js

Comment 17

[Mark Finkle (:mfinkle) (use needinfo?)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/2ba2e07e2fcd

Comment 18

[Mark Finkle (:mfinkle) (use needinfo?)]

Mozilla Metrics verified the test campaign data is being delivered to the metrics servers.

Comment 19

[Mark Finkle (:mfinkle) (use needinfo?)]

Comment on attachment 668543 [details] [diff] [review]
patch 2 v1

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 
User impact if declined: Affects marketing's ability to detemrine success of campaigns
Testing completed (on m-c, etc.): Core patch is on nightly and aurora with no regressions noticed yet.
Risk to taking this patch (and alternatives if risky): some small risk of startup speed regressions, but we are using as much async code as possible.
String or UUID changes made by this patch: none

Comment 20

[Mark Finkle (:mfinkle) (use needinfo?)]

Comment on attachment 668543 [details] [diff] [review]
patch 2 v1

Got an IRC a+ from Alex

Comment 21

[Mark Finkle (:mfinkle) (use needinfo?)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/fef8df9be7cd

Comment 22

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/2ba2e07e2fcd

Comment 23

[Aaron Train [:aaronmt]]

Should probably unpublish ReleaseSplitTest on Google Play now

Comment 24

[Mark Finkle (:mfinkle) (use needinfo?)]

(In reply to Aaron Train [:aaronmt] from comment #23)
> Should probably unpublish ReleaseSplitTest on Google Play now

Indeed. Done.