1476006 - [Shield] Pref Flip Study: Top Sites Impact on Pocket Recs / Search

Status: RESOLVED INVALID

(Shield :: Shield Study, enhancement)

Description

[tushar]

Basic description of experiment: 
We want to understand what combination of the Top Sites, Pocket Recommendations, and Web Search widgets maximize Search Queries / Pocket Recommendations clicks. 

What is the preference we will be changing? 

browser.newtabpage.activity-stream.feeds.topsites

What are the branches of the study and what values should each branch be set to? 
Control: true
Clean: false

What percentage of users do you want in each branch? 
50% Control / 50% Clean

What Channels and locales do you intend to ship to? 
Channel: Release >= 61
Locale: en-US
Geo: US

Are there specific criteria for participants? 
New users only

What is the main effect you are looking for and what data will you use to make these decisions?
Firefox retention, search volume, and clicks on Pocket Recs

Who is the owner of the data analysis for this study?
Kenny Long, with help from Shield Team

Will this experiment require uplift? 

No

QA Status of your code: 
QA Request has been submitted. 

Do you plan on surveying users at the end of the study? 

No.

Comment 1

[Ciprian Muresan [:cmuresan], Formerly of Ecosystem QA - please needinfo @mcoman for things regarding the team]

We have finished testing the Fx New Tab & Pocket Recommendations: Ordering Test.

QA’s recommendation: GREEN - SHIP IT

Reasoning:
- No issues have been found during testing.

Testing Summary:
- Full Functional test suite: TestRail;

Tested Platforms:
- Windows 10 x64
- Mac 10.13.5
- Arch Linux x64

Tested Firefox versions:
- Firefox Beta 61.0.1

Comment 2

[Jeff Griffiths (:canuckistani) (:⚡︎)]

(In reply to tushar from comment #0)
> Basic description of experiment: 
> We want to understand what combination of the Top Sites, Pocket
> Recommendations, and Web Search widgets maximize Search Queries / Pocket
> Recommendations clicks. 

Tushar, in the future a study that impacts the UI of Firefox needs to be reviewed by Firefox Product, eg Maria in this case or given that she is on PTO, escalated to Peter or myself.

Comment 3

[tushar]

(In reply to Jeff Griffiths (:canuckistani) (:⚡︎) from comment #2)
> (In reply to tushar from comment #0)
> > Basic description of experiment: 
> > We want to understand what combination of the Top Sites, Pocket
> > Recommendations, and Web Search widgets maximize Search Queries / Pocket
> > Recommendations clicks. 
> 
> Tushar, in the future a study that impacts the UI of Firefox needs to be
> reviewed by Firefox Product, eg Maria in this case or given that she is on
> PTO, escalated to Peter or myself.

Jeff - understood, will make sure that happens for any future studies.

Comment 4

[tushar]

After discussing internally, we are going to deprioritize moving forward with this study in favor of some other higher-priority studies in flight and will revisit spinning back up in a few weeks.

Comment 5

[Ed Lee :Mardak]

(In reply to tushar from comment #0)
> What is the preference we will be changing? 
> browser.newtabpage.activity-stream.feeds.topsites
> 
> What are the branches of the study and what values should each branch be set
> to? 
> Control: true
> Clean: false

For peer review of the code / behavior changes, this study should be good. This preference is the same that users normally change from multiple UIs (about:preferences or directly from about:newtab -> … section menu -> Remove Section), and both true and false are already covered by automated tests.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Getting this off the radar for 61 since it doesn't sound like we're moving forward with the study within that timeframe.