1410535 - [Shield] Pref Flip Study: Activity Stream in 57 Beta

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Jared Lockhart [:jlockhart]]

Basic description of experiment: 

Disable Activity Stream for a subset of the beta population

What is the preference we will be changing? 

browser.newtabpage.activity-stream.enabled

What are the branches of the study and what values should each branch be set to? 

Control: false
Variant: true

What percentage of users do you want in each branch? 

25% Overall
1 Control: 1 Variant

What Channels and locales do you intend to ship to? 

57 Beta all locales

What is your intended go live date and how long will the study run? 

Start Date: October 20th 2017
End Date: November 13th 2017

Are there specific criteria for participants? 

No

What is the main effect you are looking for and what data will you use to make these decisions? 

We are looking to see improvements in:
Firefox Engagement
Firefox Retention
Pocket Engagement

And not see any regressions in:
Performance

To analyze these we will be using a combination of the Experiment Telemetry provided by the main ping and custom telemetry provided by Ping Centre

Who is the owner of the data analysis for this study? 

Marina Samuel (emtwo)

Will this experiment require uplift? 

No

QA Status of your code: 

Green.  This code has been rigorously tested in Nightly and Beta.

Do you plan on surveying users at the end of the study? 

Yes.

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:

https://github.com/mozilla/activity-stream

Previous studies:
https://docs.google.com/document/d/1LxpZe7W0bZ4--fogIv1oIct1HbJ4vkcd7EuLNdWTn08/edit?usp=sharing

Comment 1

[Jared Lockhart [:jlockhart]]

Recipe slug:

pref-flip-activity-stream-57-beta-enabled-bug-1410535

Comment 2

[Matt Grimes [:Matt_G]]

Changing population sizes. This experiment will target 40% of the Beta population with an even distribution across branches.

Comment 3

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

I am wondering about the QA status. Has this (AS off with a pref) code been tested in Nightly? I know AS (on by default) code has been tested a lot in Nightly. Could you please comment on that?

Comment 4

[Tim Spurway [:tspurway]]

We are doing a try build now for AS preffed off:

https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&fromchange=63366030a0d8c261485d8896e8cf6c75e4dd3dc1&tochange=a21703bdb54e6876f09e27d62a259e932dd5b531

Comment 5

[Tim Spurway [:tspurway]]

Let's go ahead with the study.  Treeherder shows mostly green and all manual QA for Tiles functionality are checking out.  

Erin, Ritu - do you folks need anything else from us?

Comment 6

[Erin Lancaster [:elan]]

Per Matt G. we just need one final R+ from Ritu.

Comment 7

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

If QA sign off is green, please go ahead and kick off this study.

Comment 8

[Jared Lockhart [:jlockhart]]

The updated definition of this study is:

Pref: browser.newtabpage.activity-stream.enabled
slug: pref-flip-activity-stream-58-nightly-pocket-personalization-bug-1400890
Firefox Version: 57 Beta
Start Date: Oct 24
End Date: Nov 13
Percentage: 100%
Control ratio: 1
Variant ratio: 1
Control value: false
Variant value: true
Locales: All
Geos: All
Filters: Exclude the following studies:
* pref-flip-activity-stream-57-beta-snippet-1409201
* pref-flip-activity-stream-56-beta-about-home-bug-1405334
Objectives: Measure the overall impact of Activity Stream on 57 Beta engagement and retention.

Comment 9

[Jared Lockhart [:jlockhart]]

Note, in the previous comment I accidentally said the wrong slug, the slug is:

pref-flip-activity-stream-57-beta-enabled-bug-1410535

Comment 10

[Matt Grimes [:Matt_G]]

@jkerim Now that this study has ended can you please recap the outcome here and close the bug? A couple sentences is fine.

Comment 11

[:shell escalante]

closing all experiment bugs - for experiments that have ended.