[Shield] Pref Flip Study: Activity Stream 56 Beta Study

NEW
Assigned to

Status

Shield
Shield Study
11 months ago
5 months ago

People

(Reporter: jkerim, Assigned: Maria Popova, NeedInfo)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

11 months ago
This will be a multi part study, the first part is defined here, when this study is complete I will update this ticket to add the next study definition.  A link at the bottom contains a Google Doc with all of the studies.

Basic description of experiment: 

Enable the new Activity Stream new tab page for a subset of users in Firefox 56 Beta for a fixed period of time to gather telemetry data about its performance and usage  

What is the preference we will be changing? 

browser.newtabpage.activity-stream.enabled

What are the branches of the study and what values should each branch be set to? 

1 branch set to ‘true’

What percentage of users do you want in each branch? 

10% overall with a single variant which is the pref set to true

What Channels and locales do you intend to ship to? 

Firefox Versions: 56
Channels: Beta
Locales: Geo US

What is your intended go live date and how long will the study run? 

Start Date: August 17th, 2017
End Date: August 31st, 2017

Are there specific criteria for participants? 

What is the main effect you are looking for and what data will you use to make these decisions? 

We are looking to see that users interact with the elements on the Activity Stream new tab page

We are collecting telemetry using the Ping Centre telemetry system to measure and validate user interactions

Who is the owner of the data analysis for this study? 

Marina Samuel

Will this experiment require uplift? 

Yes uplift is required

QA Status of your code: 
Green, yellow, red. Your code should be QA’d to ensure that changing the preference values has the intended effect you are looking for and does not cause obvious regressions to Firefox. All experiments must pass QA. Depending on the channel/population size a dev QA may be accepted.

Do you plan on surveying users at the end of the study? 

Yes we are designing a survey found here:
https://docs.google.com/a/mozilla.com/document/d/16NtcApTE0Y2sR5uPYqjO8XfSoVztGJ-5js4TjJlMT3M/edit?usp=sharing

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:

Activity Stream 56 Beta Studies Plan:
https://docs.google.com/a/mozilla.com/document/d/1wizj95pDfb4YTc2kDxTJn5TIm7SwGetAsn1e0OgeOh4/edit?usp=sharing

Comment 1

10 months ago
It's likely that we will not be ready to start this experiment on Aug 17th as we're waiting to uplift some more changes to the system addon which we will request on Aug 17th.

We'll give an update when we have a new start date.

Comment 2

10 months ago
No problem. Just keep us posted.

Comment 3

10 months ago
Study is now live on Beta 56 for 10% of users in GEO US.
(Reporter)

Comment 4

10 months ago
Please remove the GEO US restriction and keep it at 10% of 56 Beta en-US

Comment 5

10 months ago
Done. We've removed the Geo restriction as requested.
(Reporter)

Comment 6

10 months ago
Given the information we've gathered so far, we've decided to close this experiment and begin the next one.

The next experiment definition is:

Pref: browser.newtabpage.activity-stream.enabled
Firefox Version: 56 Beta
Start Date: Aug 28
Percentage: 50%
Control ratio: 1
Variant ratio: 10
Locales: en-US
Objectives: Performance, Stability, Engagement

The PHD is the same, the only changes between the last one and this one are increasing the percentage size from 10% to 50%.
(Reporter)

Comment 7

10 months ago
Okay scratch that definition. This is the definition we want (we want a 1:1 split between control and variant):


Pref: browser.newtabpage.activity-stream.enabled
Firefox Version: 56 Beta
Start Date: Aug 29
Percentage: 50%
Control ratio: 1
Variant ratio: 1
Locales: en-US
Objectives: Performance, Stability, Engagement


I am sending an email to release management now with this proposal.
Continuing the same experiment with more of the population on beta sounds fine. Is there a planned end date?
Flags: needinfo?(jkerim)

Comment 9

10 months ago
We've pushed the change as requested. 50% of Beta 56 in en_US with a 1/1 ratio of control/treatment.
(Reporter)

Comment 10

10 months ago
In addition to the current study that is running now and will run until the end of 56 Beta, which is defined by:

Pref: browser.newtabpage.activity-stream.enabled
Firefox Version: 56 Beta
Start Date: Aug 29
Percentage: 50%
Control ratio: 1
Variant ratio: 1
Locales: en-US
Objectives: Performance, Stability, Engagement


Let's start a new study defined as follows:

Pref: browser.newtabpage.activity-stream.enabled
Firefox Version: 56 Beta
Start Date: Sept 8
Percentage: 50%
Control ratio: 1
Variant ratio: 1
Locales: *
Geos: PL, RU, GB, FR
Objectives: Performance, Stability, Engagement, Localization


Both of these studies will run until the end of the 56 beta period to allow collecting as much data as possible.

There is still a third study which has not yet launched which is specifically for Germany and depends on Pocket completing their German localized Pocket trending feed.
Flags: needinfo?(jkerim)

Comment 11

10 months ago
New geos are now running.
(Reporter)

Comment 12

9 months ago
We're starting an additional study today, which marks the last of the Activity Stream 56 Beta studies.


Pref: browser.newtabpage.activity-stream.enabled
Firefox Version: 56 Beta
Start Date: Sept 12
Percentage: 50%
Control ratio: 1
Variant ratio: 1
Locales: de-de
Geos: DE (Germany)
Objectives: Pocket Content Localization
New recipe has been pushed live for 50% of de_DE users in geo DE on beta 56. Split 1/1 treatment to control.
To work around bug 1399936, we've disabled the existing 3 recipes and recreated them. The new experiment slugs are identical except that they have "-round2" appended to the end (as do the recipe names in the admin).
I believe all of these experiments are set to end tomorrow. Is that still the case?
Flags: needinfo?(jkerim)
(Reporter)

Comment 16

9 months ago
(In reply to Matt Grimes [:Matt_G] from comment #15)
> I believe all of these experiments are set to end tomorrow. Is that still
> the case?

Yes, they should all expire at the end of 56 beta.  They are technically still running now in 57 beta and accidentally disabling Activity Stream.  Let's close all of the experiments managed by this bug:


Pref: browser.newtabpage.activity-stream.enabled
Firefox Version: 56 Beta
Start Date: Aug 29
End date: Sept 27
Percentage: 50%
Control ratio: 1
Variant ratio: 1
Locales: en-US
Objectives: Performance, Stability, Engagement

Pref: browser.newtabpage.activity-stream.enabled
Firefox Version: 56 Beta
Start Date: Sept 8
End date: Sept 27
Percentage: 50%
Control ratio: 1
Variant ratio: 1
Locales: *
Geos: PL, RU, GB, FR
Objectives: Performance, Stability, Engagement, Localization

Pref: browser.newtabpage.activity-stream.enabled
Firefox Version: 56 Beta
Start Date: Sept 12
End date: Sept 27
Percentage: 50%
Control ratio: 1
Variant ratio: 1
Locales: de-de
Geos: DE (Germany)
Objectives: Pocket Content Localization


Thanks!
Flags: needinfo?(jkerim)
Done. Users should enenroll the next time they hit the normandy service.
Hey folks. Can we get a quick recap of the results and then have this bug closed? A couple sentences is fine.
Flags: needinfo?(jkerim)
You need to log in before you can comment on or make changes to this bug.