Closed Bug 1232777 Opened 9 years ago Closed 8 years ago

Investigate using Adjust to correlate A/B experiments with retention and engagement

Categories

(Firefox for Android Graveyard :: General, defect)

Product:
Firefox for Android Graveyard
Component:
General
Version:
35 Branch
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: Margaret, Assigned: mcomella)

References

Details

(In reply to Michael Comella (:mcomella) from comment #1)
> It looks like we use FHR to do this by adding the experiment to our list of
> addons as a dummy add-on [1].
> 
> [1]:
> https://dxr.mozilla.org/mozilla-central/rev/
> cb66ffeb6725e8344818e8e2f707ae2eaeb953b4/mobile/android/base/java/org/
> mozilla/gecko/health/BrowserHealthRecorder.java#647

Yep and this is going away.
re the link in comment 0, it looks like we have four levels of hierarchy for Adjust urls:
 Network
  Campaign
   Adgroup
    Creative

Network is the only attribute that must be created manually on the website, so this likely won't work for us. Which remaining level we want to use is up to the way we currently structure our data and how we want our A/B testing to be involved in this. I'm assuming we'll want to see all the installs for a particular ad group of a campaign no matter which A/B test the user is on, so I'd take it we'd want to use the Creative level.

It seems that each level is defined as a single String so if we were to use it, we'd have to prefix/affix the experiment to the Creative String. However, this further divides the Creative value, so it may make it difficult (e.g. analyzer needs to click two boxes for the dashboard) or impossible (previously mentioned feature does not exist) for an analyzer to use the Creative field for whatever purpose it's being used for now.

Finkle, do you know what's going on with the current use of these values or who I should ask?

Note: I'm still unclear where these urls are created and how we could modify them.
Flags: needinfo?(mark.finkle)
Yeah, I have no idea how to use these strings yet. We asked Adjust to give us some additional hints. We could also try to play in the Adjust sandbox mode and see what gets sent from a local build.

Bug 1215366 has links to some protected CSV files
Flags: needinfo?(mark.finkle)
Here is the type of hierarchy I could see us using it:
Network = "android_ab_tests"
Campaign = "2015-12-16_onboarding"
Adgroup = "control" / "version_a" / "version_b" / "version_c"
creative = (optional)

I setup the email marketing team in a similar fashion so that they could track generated installs. We log all engagement and product marketing URL hierarchy here: https://docs.google.com/spreadsheets/d/151_fWwEPxmPN_AbHCp-sqAkshDCLaABRrVaowxoFS10/edit#gid=0

In our case, all our tests fall under their own "network" isolated from all the campaign data. We can easily see all variations (adgroup) within a test (campaign). For the most part, I think "creative" is not needed.

The URLs can be made on the fly. No need to add them within Adjust. As soon as we pass URLs with new parameters, Adjust saves them as is.

I hope this helps.
I have one small concern. (which might be nothing)

Are we sure that we won't be overwriting campaign data that might already be tracked via a paid install?
Or will Adjust track our real campaign AND our fake A/B test campaign.
(In reply to Alex Davis [:adavis] from comment #6)
> Are we sure that we won't be overwriting campaign data that might already be
> tracked via a paid install?
> Or will Adjust track our real campaign AND our fake A/B test campaign.

Shucks – I was just about to ask the same question! I suppose we can test this in the sandbox.

To add a bit of clarifying info, with Alex's scheme in comment 5, I interpreted that users would send one referrer url for the ad campaign data and one for the A/B test data. (My scheme in comment 3 was assuming we could not send multiple referrer urls)
Assignee: nobody → michael.l.comella
The SDK readme on github does not indicate how to set these values, however, the documentation implies the there can only be one "attribution" (or referrer uri) per install (item 15):

You can register a listener to be notified of tracker attribution changes. [ed: Okay, "changes", plural, but we're contradicted later]

The listener function will be called when the SDK receives the final attribution information. [ed: final attribution information] Within the listener function you have access to the attribution parameter. Here is a quick summary of its properties:

    String trackerToken the tracker token of the current install.
---

Note that is *the* tracker token of the current install.

Later, at [1]:

If you use this facility, please note that in-app attribution data access is designed to forward the data to you, in a similar manner as server-to-server callbacks.
---

My interpretation is that these values are set via the web SDK (I'm assuming there is one), stored on their servers, and unchangeable via the Android application itself.

Finkle, does this seem plausible? I'll dig into the source to see if I can find out anything else useful.

If we were stuck with this setup, we could add a randomly generated ID to the "Creative" field (or maybe telemetry, somewhere) and clients can bucket themselves via a local algorithm. It's not as good as the existing solution, as I understand it, but it could work...

[1]: https://github.com/adjust/sdks/blob/master/doc/attribution-data.md
Flags: needinfo?(mark.finkle)
Oh, or maybe the dashboards can give us enough info with these `AdjustEvent`s.
It seems like this path is leading us into the Weeds™️. Let's put a pause on this route and see if we can find a different approach. Bug 1205835 might be the simplest, easiest way forward for getting retention and experiments. Yes, it might take some time to get it going, but it doesn't make sense to create an awful hack with Adjust.
Flags: needinfo?(mark.finkle)
We spoke during the front-end team meeting – I'm going to look into these `AdjustEvent`s now that I have access to the sandbox dashboard and can determine if we can link those with retention data. If that doesn't pan out, I'll work on implementing Telemetry.
I got close – I can seemingly filter the dashboard by "Users that triggered" custom `AdjustEvent`s but when I do so, the "Retained Users" field changes to "Converted Users" and I'm not quite sure what that means. I sent an email to adjust support to clarify and see if:
 * We can get "Retained Users" w/ these custom `AdjustEvent`s
 * If not, to find out what the "Converted Users" field is and if it's is sufficient for retention
 * To get an example of what this dashboard looks like (since we don't have custom events, we have no data, and the dashboard is not shown. This dashboard does not work in sandbox mode)
 * Any alternative ways they might suggest implementing this.

I thought of another other approach with Adjust's attribution/event callbacks but it seems inferior to writing our own Telemetry uploader. This approach was to use the callbacks to send the Android application a callback when the retention event is fired (if there is such an event). In this callback, we could add in our A/B test buckets, forward the data to our own servers and create our own dashboard. However, there are some unknowns – if the callback does not complete (e.g. user closes the application), does Adjust resend the ping? afaik these events are not synchronous – how long is it before Adjust sends the ping?

Given the unknowns and the fact that this implementation would require us to contact our own servers, it'd probably be easier to write the simple Telemetry uploader and free ourselves from the Adjust dependency. This is especially true given we've (afaik) already created the Telemetry protoctols and have the infrastructure to process the output.
(In reply to Michael Comella (:mcomella) from comment #12)
> I got close – I can seemingly filter the dashboard by "Users that triggered"
> custom `AdjustEvent`s but when I do so, the "Retained Users" field changes
> to "Converted Users" and I'm not quite sure what that means. I sent an email
> to adjust support to clarify and see if:
>  * We can get "Retained Users" w/ these custom `AdjustEvent`s
>  * If not, to find out what the "Converted Users" field is and if it's is
> sufficient for retention
>  * To get an example of what this dashboard looks like (since we don't have
> custom events, we have no data, and the dashboard is not shown. This
> dashboard does not work in sandbox mode)
>  * Any alternative ways they might suggest implementing this.
> 
> I thought of another other approach with Adjust's attribution/event
> callbacks but it seems inferior to writing our own Telemetry uploader. This
> approach was to use the callbacks to send the Android application a callback
> when the retention event is fired (if there is such an event). In this
> callback, we could add in our A/B test buckets, forward the data to our own
> servers and create our own dashboard. However, there are some unknowns – if
> the callback does not complete (e.g. user closes the application), does
> Adjust resend the ping? afaik these events are not synchronous – how long is
> it before Adjust sends the ping?
> 
> Given the unknowns and the fact that this implementation would require us to
> contact our own servers, it'd probably be easier to write the simple
> Telemetry uploader and free ourselves from the Adjust dependency. This is
> especially true given we've (afaik) already created the Telemetry protoctols
> and have the infrastructure to process the output.

Yes, uploading pings that are processed by the existing telemetry pipeline would be easier to handle from the server side.

Using Adjust for A/B testing seemed like a nice option if we could get it mostly for free, but it sounds like we can't, so I think we should move forward with building our own telemetry uploader. The structure of these pings can be really simple to start, so this shouldn't be too hard. Or rather, the challenges will be in just doing the work to implement it, rather than conceptual challenges trying to understand a system we don't control.
We're moving forward with telemetry (bug 1205835) which will later include A/B testing experiments.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.