1405332 - [Shield] Pref Flip Study: Activity Stream in 56 Release

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Jared Lockhart [:jlockhart]]

Basic description of experiment: 

Enable Activity Stream for a subset of the release population

What is the preference we will be changing? 

browser.newtabpage.activity-stream.enabled

What are the branches of the study and what values should each branch be set to? 

Control: false
Variant: true

What percentage of users do you want in each branch? 

1% Overall
1 Control: 1 Variant

What Channels and locales do you intend to ship to? 

56 Release all locales
What is your intended go live date and how long will the study run? 

Start Date: October 2nd 2017
End Date: November 13th 2017

Are there specific criteria for participants? 

No

What is the main effect you are looking for and what data will you use to make these decisions? 

We are looking to see improvements in:
Firefox Engagement
Firefox Retention
Pocket Engagement

And not see any regressions in:
Performance

To analyze these we will be using a combination of the Experiment Telemetry provided by the main ping and custom telemetry provided by Ping Centre

Who is the owner of the data analysis for this study? 

Marina Samuel (emtwo)

Will this experiment require uplift? 

No

QA Status of your code: 

Green.  This code has been rigorously tested in Nightly and Beta.

Do you plan on surveying users at the end of the study? 

Yes.

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:

https://github.com/mozilla/activity-stream

Previous studies:
https://docs.google.com/document/d/1LxpZe7W0bZ4--fogIv1oIct1HbJ4vkcd7EuLNdWTn08/edit?usp=sharing

Comment 1

[Jared Lockhart [:jlockhart]]

Study PHD:

https://docs.google.com/a/mozilla.com/document/d/1RCWJ18hoXEGKh_9R3u7wBSDu02zbNewXW3VnWjMxea0/edit?usp=sharing

Comment 2

[Matt Grimes [:Matt_G]]

Looks good to me. r+

Comment 3

[Jared Lockhart [:jlockhart]]

Recipe slug:

pref-flip-activity-stream-56-release-bug-1405332

Comment 4

[Liz Henry (:lizzard) (relman/hg->git project)]

OK, please feel free to go ahead on 56. But, keep in mind we have not enabled 56.0 release to 100% of users yet and that may not happen until tomorrow morning (Wednesday ) when you look at your uptake data.

Comment 5

[Robert (rrayborn, he/him)]

matt_g launched. jgaunt reviewed.

Comment 6

[Peter Dolanjski [:pdol]]

Looking at the population graph here: https://sql.telemetry.mozilla.org/queries/47783#129215 makes me a little concerned about retention rates.  There seems to be a growing delta between the population sizes.  This may just be differences in enrollment rates, but I just wanted to make a note here so we can look at retention rates.  Since the subsequent graphs are scaled per user, they may be hiding the users we no longer see dropping off.

Comment 7

[Jared Lockhart [:jlockhart]]

Based on the fact that the Activity Stream experience that was shipped in 56 release is lacking a significant number of the design, functionality, and performance enhancements that are present in the more recent version that appears in 57, our conclusion is that this study shows us that the engagement and retention regression we measured here is a result of the impact of the system addon being present without all of the benefits it is meant to provide, as a result we see net negative engagement and retention regressions.  This is not representative of the impact that Activity Stream will have in 57 release, and so I think we should close this study and shift our attention to a similar study in 57 Beta that will study the same effect (Activity Stream enabled/disabled) to give us a clearer picture of how the version that ships in 57 overall impacts engagement and retention.

That study can be found here:

https://bugzilla.mozilla.org/show_bug.cgi?id=1410535

Comment 8

[Jared Lockhart [:jlockhart]]

This experiment can be closed.

Comment 9

[Matt Grimes [:Matt_G]]

The experiment has been disabled. Let me know if you need anything else!

Comment 10

[Marina Samuel [:emtwo]]

Experiment Dashboard: https://sql.telemetry.mozilla.org/dashboard/activity-stream-system-addon-experiment-v1-release

We also wanted to learn about pocket usage from this experiment. A couple of charts with pocket click rates, impression and click counts are included in the dashboard above. A couple of notes are:

1) Since the experiment had poor retention and engagement overall, these interaction rates are also lower than we expect in 57 release

2) There is a bug where impression counts are not available in 56 release.

We will revisit the pocket results in the 57 beta study as well (https://bugzilla.mozilla.org/show_bug.cgi?id=1410535) where there is no missing impressions bug.