1301424 - [SHIELD] Data review for Activity Stream Validation Experiment

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Tim Spurway [:tspurway]]

data-review: r? :rweiss

The data collected by Activity Stream is detailed here:

https://github.com/mozilla/activity-stream/blob/master/data_dictionary.md

and here:

https://github.com/mozilla/activity-stream/blob/master/data_events.md

In addition, the Shield addon will include a patch to the existing about:newtab code to alter the pings it sends in the following way:

- include a `client_id` field (unique browser identifier)
- add an `action` field to indicate that the ping is originating from the Tiles control group
- add a ping for session termination 

In all other respects, the ping from the Tiles branch of the experiment will be exactly the same as the Activity Stream branch.  This will allow us to compare the two variants.

The study should run at least one month (for MAU calcs).

:emtwo is the owner of the data analysis for this experiment.

The experiment data will be analysed to compare various engagement and retention metrics.  It would seem rational to compare clicks on 'frecent' Tiles with clicks on Activity Stream top sites, to compare searchbox usage in both variants, as well as comparing distributions of invocations of about:newtab for each branch.

Comment 1

[[e-mail]]

Two questions and I will sign off:

1) It is fair to say that the ultimate goal is to understand if Activity Stream encourages greater clickthrough to top sites compared to frecent, more searchbox usage, and number of newtabs opened, correct?

2) client_id is the UT client id or a different client id?

Comment 2

[Tim Spurway [:tspurway]]

1) Yes.  In addition, I think we need to look at Firefox retention for the two different groups as well.  We might also consider all click interactions on the Activity Stream vs. clicks on Tiles.  Another consideration is to compare 'unique users who clicked' instead of just a CTR metric that can be skewed by eager clickers.

2) Yeah, Activity Stream uses UT client id - it's the same one we want to use for the Tiles variant

Comment 3

[Tim Spurway [:tspurway]]

(In reply to Rebecca Weiss from comment #1)

Just to clarify our expectations from the experiment here.

> Two questions and I will sign off:
> 
> 1) It is fair to say that the ultimate goal is to understand if Activity
> Stream encourages greater clickthrough to top sites compared to frecent,
> more searchbox usage, and number of newtabs opened, correct?
> 

Yes.  We originally sought to make retention a key part of the experiment, but feel that would be very difficult considering the time periods and other factors we have discussed.  The study will focus on a Activity Stream / Tiles comparison of clickthrough on top sites/frecent, searchbox, and newtabs opened.

r? :rweiss

Comment 4

[[e-mail]]

data-review signoff!

Comment 5

[Tim Spurway [:tspurway]]

The results of our last Shield experiment are here:

https://sql.telemetry.mozilla.org/dashboard/activity-stream-a-b-testing-shield-study

We are going to re-run the study with the same variants and parameters.

r? :rweiss

Comment 6

[[e-mail]]

r+