Basic description of experiment: Add a second row of topsites to Activity Stream users in 57 Beta. What is the preference we will be changing? browser.newtabpage.activity-stream.topSitesCount What are the branches of the study and what values should each branch be set to? Control: 6 Variant: 12 What percentage of users do you want in each branch? 25% Overall 1 Control: 1 Variant What Channels and locales do you intend to ship to? 57 Beta Locale: en-US Geo: US What is your intended go live date and how long will the study run? Start Date: October 25th 2017 End Date: November 13th 2017 Are there specific criteria for participants? The following pref must be set to this value: browser.newtabpage.activity-stream.enabled;true What is the main effect you are looking for and what data will you use to make these decisions? Past experiments have shown that two rows of topsites significantly increases engagement with Firefox. We will validate this using both UT and PC data to show that engagement is increased and has a positive effect on browser usage. Who is the owner of the data analysis for this study? Marina Samuel (emtwo) Will this experiment require uplift? No QA Status of your code: Green. This code has been rigorously tested in Nightly and Beta. Do you plan on surveying users at the end of the study? No. Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists: This study supports an existing study of Activity Stream in 57 beta: https://bugzilla.mozilla.org/show_bug.cgi?id=1410535 https://github.com/mozilla/activity-stream Previous studies: https://docs.google.com/document/d/1LxpZe7W0bZ4--fogIv1oIct1HbJ4vkcd7EuLNdWTn08/edit?usp=sharing
Recipe slug: pref-flip-activity-stream-57-beta-two-rows-bug-1411695
I am launching too many studies at the same time and am confusing populations. This study should not have locale/geo filtering. The correct definition is: pref: browser.newtabpage.activity-stream.topSitesCount slug: pref-flip-activity-stream-57-beta-two-rows-bug-1411695 Firefox version: 57 Beta Start Date: Oct 25 2017 End Date: Nov 13 2017 Percentage: 25% Control Ratio: 1 Variant Ratio: 1 Control Value: 6 Variant Value: 12 Locales: All Geos: All Filters: browser.newtabpage.activity-stream.enabled=true
Please go ahead with this study on beta57. This is what the distribution of users for both Activity Stream studies will look like: Study 1: * 50% AS off * 50% AS on Study 2: * 50% AS off * 12.5% AS on 2 rows of topsites * 37.5% AS on 1 row of topsites
Recipe is live: https://normandy.cdn.mozilla.net/api/v2/recipe/321/ We were a bit confused about what sample was wanted. We went with the more conservative sample, but it would be trivial to double it. Right now we're doing an even split of 25% of the AS-enabled population. That makes our overall percentages a bit different than Ritu articulated. With both Study 1 & 2 enabled we have: 50% AS-off 37.5% AS-on with default topSiteCount 6.25% AS-on with control (6) topSiteCount 6.25% AS-on with variant (12) topSiteCount If I'm incorrect and you wanted 12.5%/12.5% for those last 2 rows we can change it without changing the enrollment of the existing users. Thanks
Rob, Yes your calculation is correct and is exactly what we want. However the recipe that launched was misconfigured (my fault!) so we need to close that recipe and launch a new one. I forgot to specify that this study needs to set a USER pref branch rather than a DEFAULT pref branch. It should look like this: slug: pref-flip-activity-stream-57-beta-two-rows-user-pref-bug-1411695 pref: browser.newtabpage.activity-stream.topSitesCount pref-branch: User Firefox version: 57 Beta Start Date: Oct 30 2017 End Date: Nov 13 2017 Percentage: 25% Control Ratio: 1 Variant Ratio: 1 Control Value: 6 Variant Value: 12 Locales: All Geos: All Filters: browser.newtabpage.activity-stream.enabled=true Sorry for the mixup and thanks for the help!
I've closed the original study and the new one is live. Let me know if you need anything else!