1411695 - [Shield] Pref Flip Study: Activity Stream Two Rows of Topsites in 57 Beta

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Jared Lockhart [:jlockhart]]

Basic description of experiment: 

Add a second row of topsites to Activity Stream users in 57 Beta.

What is the preference we will be changing? 

browser.newtabpage.activity-stream.topSitesCount

What are the branches of the study and what values should each branch be set to? 

Control: 6
Variant: 12

What percentage of users do you want in each branch? 

25% Overall
1 Control: 1 Variant

What Channels and locales do you intend to ship to? 

57 Beta
Locale: en-US
Geo: US

What is your intended go live date and how long will the study run? 

Start Date: October 25th 2017
End Date: November 13th 2017

Are there specific criteria for participants? 

The following pref must be set to this value:

browser.newtabpage.activity-stream.enabled;true

What is the main effect you are looking for and what data will you use to make these decisions? 

Past experiments have shown that two rows of topsites significantly increases engagement with Firefox.

We will validate this using both UT and PC data to show that engagement is increased and has a positive effect on browser usage.

Who is the owner of the data analysis for this study? 

Marina Samuel (emtwo)

Will this experiment require uplift? 

No

QA Status of your code: 

Green.  This code has been rigorously tested in Nightly and Beta.

Do you plan on surveying users at the end of the study? 

No.

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:

This study supports an existing study of Activity Stream in 57 beta:
https://bugzilla.mozilla.org/show_bug.cgi?id=1410535

https://github.com/mozilla/activity-stream

Previous studies:
https://docs.google.com/document/d/1LxpZe7W0bZ4--fogIv1oIct1HbJ4vkcd7EuLNdWTn08/edit?usp=sharing

Comment 1

[Jared Lockhart [:jlockhart]]

Recipe slug:

pref-flip-activity-stream-57-beta-two-rows-bug-1411695

Comment 2

[Jared Lockhart [:jlockhart]]

I am launching too many studies at the same time and am confusing populations.  This study should not have locale/geo filtering.  The correct definition is:

pref: browser.newtabpage.activity-stream.topSitesCount
slug: pref-flip-activity-stream-57-beta-two-rows-bug-1411695
Firefox version: 57 Beta
Start Date: Oct 25 2017
End Date: Nov 13 2017
Percentage: 25%
Control Ratio: 1
Variant Ratio: 1
Control Value: 6
Variant Value: 12
Locales: All
Geos: All
Filters: browser.newtabpage.activity-stream.enabled=true

Comment 3

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Please go ahead with this study on beta57.

This is what the distribution of users for both Activity Stream studies will look like:

Study 1:
 * 50% AS off
 * 50% AS on

Study 2:
 * 50% AS off
 * 12.5% AS on 2 rows of topsites
  * 37.5% AS on 1 row of topsites

Comment 4

[Robert (rrayborn, he/him)]

Recipe is live: https://normandy.cdn.mozilla.net/api/v2/recipe/321/

We were a bit confused about what sample was wanted.  We went with the more conservative sample, but it would be trivial to double it.

Right now we're doing an even split of 25% of the AS-enabled population.  That makes our overall percentages a bit different than Ritu articulated.

With both Study 1 & 2 enabled we have:
50% AS-off
37.5% AS-on with default topSiteCount
6.25% AS-on with control (6) topSiteCount
6.25% AS-on with variant (12) topSiteCount

If I'm incorrect and you wanted 12.5%/12.5% for those last 2 rows we can change it without changing the enrollment of the existing users.

Thanks

Comment 5

[Jared Lockhart [:jlockhart]]

Rob,

Yes your calculation is correct and is exactly what we want.  However the recipe that launched was misconfigured (my fault!) so we need to close that recipe and launch a new one.  I forgot to specify that this study needs to set a USER pref branch rather than a DEFAULT pref branch. 

It should look like this:

slug: pref-flip-activity-stream-57-beta-two-rows-user-pref-bug-1411695
pref: browser.newtabpage.activity-stream.topSitesCount
pref-branch: User
Firefox version: 57 Beta
Start Date: Oct 30 2017
End Date: Nov 13 2017
Percentage: 25%
Control Ratio: 1
Variant Ratio: 1
Control Value: 6
Variant Value: 12
Locales: All
Geos: All
Filters: browser.newtabpage.activity-stream.enabled=true


Sorry for the mixup and thanks for the help!

Comment 6

[Matt Grimes [:Matt_G]]

I've closed the original study and the new one is live. Let me know if you need anything else!

Comment 7

[Matt Grimes [:Matt_G]]

@jkerim what was the outcome? Can you please update this bug and then close it?

Comment 8

[:shell escalante]

closing all experiment bugs - for experiments that have ended.