Closed
Bug 1389722
Opened 7 years ago
Closed 6 years ago
[Shield] Pref Flip Study: Activity Stream 56 Beta Study
Categories
(Shield :: Shield Study, enhancement)
Shield
Shield Study
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: jlockhart, Assigned: mpopova, NeedInfo)
Details
This will be a multi part study, the first part is defined here, when this study is complete I will update this ticket to add the next study definition. A link at the bottom contains a Google Doc with all of the studies. Basic description of experiment: Enable the new Activity Stream new tab page for a subset of users in Firefox 56 Beta for a fixed period of time to gather telemetry data about its performance and usage What is the preference we will be changing? browser.newtabpage.activity-stream.enabled What are the branches of the study and what values should each branch be set to? 1 branch set to ‘true’ What percentage of users do you want in each branch? 10% overall with a single variant which is the pref set to true What Channels and locales do you intend to ship to? Firefox Versions: 56 Channels: Beta Locales: Geo US What is your intended go live date and how long will the study run? Start Date: August 17th, 2017 End Date: August 31st, 2017 Are there specific criteria for participants? What is the main effect you are looking for and what data will you use to make these decisions? We are looking to see that users interact with the elements on the Activity Stream new tab page We are collecting telemetry using the Ping Centre telemetry system to measure and validate user interactions Who is the owner of the data analysis for this study? Marina Samuel Will this experiment require uplift? Yes uplift is required QA Status of your code: Green, yellow, red. Your code should be QA’d to ensure that changing the preference values has the intended effect you are looking for and does not cause obvious regressions to Firefox. All experiments must pass QA. Depending on the channel/population size a dev QA may be accepted. Do you plan on surveying users at the end of the study? Yes we are designing a survey found here: https://docs.google.com/a/mozilla.com/document/d/16NtcApTE0Y2sR5uPYqjO8XfSoVztGJ-5js4TjJlMT3M/edit?usp=sharing Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists: Activity Stream 56 Beta Studies Plan: https://docs.google.com/a/mozilla.com/document/d/1wizj95pDfb4YTc2kDxTJn5TIm7SwGetAsn1e0OgeOh4/edit?usp=sharing
Comment 1•7 years ago
|
||
It's likely that we will not be ready to start this experiment on Aug 17th as we're waiting to uplift some more changes to the system addon which we will request on Aug 17th. We'll give an update when we have a new start date.
Comment 2•7 years ago
|
||
No problem. Just keep us posted.
Comment 3•7 years ago
|
||
Study is now live on Beta 56 for 10% of users in GEO US.
Reporter | ||
Comment 4•7 years ago
|
||
Please remove the GEO US restriction and keep it at 10% of 56 Beta en-US
Comment 5•7 years ago
|
||
Done. We've removed the Geo restriction as requested.
Reporter | ||
Comment 6•7 years ago
|
||
Given the information we've gathered so far, we've decided to close this experiment and begin the next one. The next experiment definition is: Pref: browser.newtabpage.activity-stream.enabled Firefox Version: 56 Beta Start Date: Aug 28 Percentage: 50% Control ratio: 1 Variant ratio: 10 Locales: en-US Objectives: Performance, Stability, Engagement The PHD is the same, the only changes between the last one and this one are increasing the percentage size from 10% to 50%.
Reporter | ||
Comment 7•7 years ago
|
||
Okay scratch that definition. This is the definition we want (we want a 1:1 split between control and variant): Pref: browser.newtabpage.activity-stream.enabled Firefox Version: 56 Beta Start Date: Aug 29 Percentage: 50% Control ratio: 1 Variant ratio: 1 Locales: en-US Objectives: Performance, Stability, Engagement I am sending an email to release management now with this proposal.
Comment 8•7 years ago
|
||
Continuing the same experiment with more of the population on beta sounds fine. Is there a planned end date?
Flags: needinfo?(jkerim)
Comment 9•7 years ago
|
||
We've pushed the change as requested. 50% of Beta 56 in en_US with a 1/1 ratio of control/treatment.
Reporter | ||
Comment 10•7 years ago
|
||
In addition to the current study that is running now and will run until the end of 56 Beta, which is defined by: Pref: browser.newtabpage.activity-stream.enabled Firefox Version: 56 Beta Start Date: Aug 29 Percentage: 50% Control ratio: 1 Variant ratio: 1 Locales: en-US Objectives: Performance, Stability, Engagement Let's start a new study defined as follows: Pref: browser.newtabpage.activity-stream.enabled Firefox Version: 56 Beta Start Date: Sept 8 Percentage: 50% Control ratio: 1 Variant ratio: 1 Locales: * Geos: PL, RU, GB, FR Objectives: Performance, Stability, Engagement, Localization Both of these studies will run until the end of the 56 beta period to allow collecting as much data as possible. There is still a third study which has not yet launched which is specifically for Germany and depends on Pocket completing their German localized Pocket trending feed.
Flags: needinfo?(jkerim)
Comment 11•7 years ago
|
||
New geos are now running.
Reporter | ||
Comment 12•7 years ago
|
||
We're starting an additional study today, which marks the last of the Activity Stream 56 Beta studies. Pref: browser.newtabpage.activity-stream.enabled Firefox Version: 56 Beta Start Date: Sept 12 Percentage: 50% Control ratio: 1 Variant ratio: 1 Locales: de-de Geos: DE (Germany) Objectives: Pocket Content Localization
Comment 13•7 years ago
|
||
New recipe has been pushed live for 50% of de_DE users in geo DE on beta 56. Split 1/1 treatment to control.
Comment 14•7 years ago
|
||
To work around bug 1399936, we've disabled the existing 3 recipes and recreated them. The new experiment slugs are identical except that they have "-round2" appended to the end (as do the recipe names in the admin).
Comment 15•7 years ago
|
||
I believe all of these experiments are set to end tomorrow. Is that still the case?
Flags: needinfo?(jkerim)
Reporter | ||
Comment 16•7 years ago
|
||
(In reply to Matt Grimes [:Matt_G] from comment #15) > I believe all of these experiments are set to end tomorrow. Is that still > the case? Yes, they should all expire at the end of 56 beta. They are technically still running now in 57 beta and accidentally disabling Activity Stream. Let's close all of the experiments managed by this bug: Pref: browser.newtabpage.activity-stream.enabled Firefox Version: 56 Beta Start Date: Aug 29 End date: Sept 27 Percentage: 50% Control ratio: 1 Variant ratio: 1 Locales: en-US Objectives: Performance, Stability, Engagement Pref: browser.newtabpage.activity-stream.enabled Firefox Version: 56 Beta Start Date: Sept 8 End date: Sept 27 Percentage: 50% Control ratio: 1 Variant ratio: 1 Locales: * Geos: PL, RU, GB, FR Objectives: Performance, Stability, Engagement, Localization Pref: browser.newtabpage.activity-stream.enabled Firefox Version: 56 Beta Start Date: Sept 12 End date: Sept 27 Percentage: 50% Control ratio: 1 Variant ratio: 1 Locales: de-de Geos: DE (Germany) Objectives: Pocket Content Localization Thanks!
Flags: needinfo?(jkerim)
Comment 17•7 years ago
|
||
Done. Users should enenroll the next time they hit the normandy service.
Comment 18•6 years ago
|
||
Hey folks. Can we get a quick recap of the results and then have this bug closed? A couple sentences is fine.
Flags: needinfo?(jkerim)
Assignee | ||
Updated•6 years ago
|
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•