1410483 - [Shield] Pref Flip Study: Activity Stream 58 Nightly Optimized Pocket Personalization

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Jared Lockhart [:jlockhart]]

Basic description of experiment: 

Pocket is integrated into the new tab page via Activity Stream. All users presently receive the same set of stories in the same order.  By analyzing a users history locally, we can determine a set of domains they visit frequently, and use that data to infer which stories would be more relevant to that user.  All of this processing will happen client side.  This experiment will enable this mechanism in such a way that it performs the history analysis and matching but does not affect the preference of stories the user see.  The purpose of this experiment is to determine whether this computational workload will in any way affect or regress any performance or engagement metrics.

This experiment builds on top of https://bugzilla.mozilla.org/show_bug.cgi?id=1400890 and lands performance enhancements to attempt to decrease the performance impact on users.

What is the preference we will be changing? 

browser.newtabpage.activity-stream.feeds.section.topstories.options

What are the branches of the study and what values should each branch be set to? 

Control: 

{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"learn_more_endpoint":"https://getpocket.cdn.mozilla.net/firefox_learnmore?src=ff_newtab","provider_header":"pocket_feedback_header","provider_description":"pocket_description","provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.cdn.mozilla.net/explore/trending?src=ff_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=2&consumer_key=$apiKey&locale_lang=en-US","stories_referrer":"https://getpocket.com/recommendations","info_link":"https://www.mozilla.org/privacy/firefox/#pocketstories","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":false,"personalized":false}

Personalized:

{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"learn_more_endpoint":"https://getpocket.cdn.mozilla.net/firefox_learnmore?src=ff_newtab","provider_header":"pocket_feedback_header","provider_description":"pocket_description","provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.cdn.mozilla.net/explore/trending?src=ff_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=2&consumer_key=$apiKey&locale_lang=en-US","stories_referrer":"https://getpocket.com/recommendations","info_link":"https://www.mozilla.org/privacy/firefox/#pocketstories","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":false,"personalized":true}

What percentage of users do you want in each branch? 

100% of 58 Nightly with 1 Control : 1 Personalized

What Channels and locales do you intend to ship to? 

Channel: 58 Nightly
Locale: en-US
Geo: US

What is your intended go live date and how long will the study run? 

Start: Oct 20th 2017
End: Nov 14th 2017

Are there specific criteria for participants? 

No

What is the main effect you are looking for and what data will you use to make these decisions? 

We are looking to monitor performance and engagement metrics collected by Ping Centre to monitor any regressions

Who is the owner of the data analysis for this study? 

Marina Samuel 

Will this experiment require uplift? 

No

QA Status of your code: 

The code has been reviewed and merged into mozilla central and tested locally.

Do you plan on surveying users at the end of the study? 

No

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:

None

Comment 1

[Jared Lockhart [:jlockhart]]

Experiment slug:

pref-flip-activity-stream-58-nightly-optimized-pocket-personalization-bug-1410483

Comment 2

[Jared Lockhart [:jlockhart]]

We must also remember to set:

Pref Branch: User

Comment 3

[Matt Grimes [:Matt_G]]

This study is now live on Nightly 58 for Geo US and en_US locale