1406513 - [Shield] Pref Flip Study: Activity Stream about:home in 58 Nightly

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Jared Lockhart [:jlockhart]]

Basic description of experiment: 

The first screen that appears on Firefox startup is called about:home.  Without Activity Stream, it shows a search bar and a Firefox logo and some buttons with common tasks like restoring the previous session.  Activity Stream supports the capability of replacing this screen with the new Activity Stream newtab experience, so that about:home and about:newtab both show the same Activity Stream screen.  Because this involves changing the way that the first page is rendered, it necessarily affects the overall perceived performance of the browser startup flow.  This functionality is DEFAULT ENABLED in 58 Nightly.  We would like to DISABLE it for a cohort of users so that we can gather data to understand whether this regresses any measurable performance or engagement metrics. 

What is the preference we will be changing? 

browser.newtabpage.activity-stream.aboutHome.enabled

What are the branches of the study and what values should each branch be set to? 

Control: true
Variant: false

What percentage of users do you want in each branch? 

50% Overall
1 Control: 1 Variant

What Channels and locales do you intend to ship to? 

58 Nightly all locales

What is your intended go live date and how long will the study run? 

Start Date: October 6th 2017
End Date: November 13th 2017

Are there specific criteria for participants? 

No

What is the main effect you are looking for and what data will you use to make these decisions? 

Unified Telemetry Engagement and Performance metrics
Validate that we do not regress any performance or engagement

Ping Centre Telemetry
We should see a measurable increase in Activity Stream engagement

Who is the owner of the data analysis for this study? 

Marina Samuel (emtwo)

Will this experiment require uplift? 

No

QA Status of your code: 

Green.  This code has been rigorously tested in Nightly, including side by side performance analysis with both the old about:home screen and Chrome for Firefox startup.

Do you plan on surveying users at the end of the study? 

No

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:

https://github.com/mozilla/activity-stream

Previous studies:

https://docs.google.com/document/d/1LxpZe7W0bZ4--fogIv1oIct1HbJ4vkcd7EuLNdWTn08/edit?usp=sharing

Comment 1

[Jared Lockhart [:jlockhart]]

Recipe slug:

pref-flip-activity-stream-58-nightly-about-home-bug-1406513

Comment 2

[Matt Grimes [:Matt_G]]

This experiment has ended per the email from jkerim