1432989 - [Shield] Pref Flip Study: New Tab 59 Beta Pocket Sponsored Stories

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Nate Weiner]

Basic description of experiment: 

Pocket recommendations are integrated into the new tab page via Activity Stream. For users in the test group, they would be eligible to see a sponsored story periodically in the 3rd position of the recommendations section. This experiment enables the ability for these stories to be shown as well as an additional option inside of the New Tab options pane that allows users to turn off sponsored stories if they choose to.

More details about the test and why we’re doing it is on Mozilla’s blog here: 
https://blog.mozilla.org/futurereleases/2018/01/24/update-on-content-recommendations-in-firefox-new-tab-page/

What is the preference we will be changing? 

browser.newtabpage.activity-stream.feeds.section.topstories.options

What are the branches of the study and what values should each branch be set to? 

Control: 


{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_header":"pocket_feedback_header","provider_description":"pocket_description","provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=2&consumer_key=$apiKey&locale_lang=en-US&feed_variant=beta_spocs_control","stories_referrer":"http://getpocket.com/recommendations","info_link":"https://www.mozilla.org/privacy/firefox/#pocketstories","disclaimer_link":"https://getpocket.cdn.mozilla.net/firefox/new_tab_learn_more","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":false,"personalized":true}

Test:

{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_header":"pocket_feedback_header","provider_description":"pocket_description","provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=2&consumer_key=$apiKey&locale_lang=en-US&feed_variant=beta_spocs_test","stories_referrer":"http://getpocket.com/recommendations","info_link":"https://www.mozilla.org/privacy/firefox/#pocketstories","disclaimer_link":"https://getpocket.cdn.mozilla.net/firefox/new_tab_learn_more","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":true,"personalized":true}

What percentage of users do you want in each branch? 


Control: 50%
Test: 50%
Total: 100%

What Channels and locales do you intend to ship to? 

Channel: 59 Beta
Locale: en-US
Geo: US

What is your intended go live date and how long will the study run? 

Start: Monday Jan 29th 2018
End: Monday Feb 12th 2018

Are there specific criteria for participants? 

No

What is the main effect you are looking for and what data will you use to make these decisions? 

While this code has been in Firefox since 57, we are using the time in Beta to do additional smoke tests and verify things work correctly without regression prior to a test within Release in mid February. We hope to see no regressions in any other engagement or performance metrics, either in Ping Centre or Firefox Telemetry.

Who is the owner of the data analysis for this study? 

Kenny Long

Will this experiment require uplift? 

No

QA Status of your code: 

The primary mechanisms of this code have been live and tested in Firefox Release. This change only alters a URL parameter when data is fetched from the server and enables an additional option in settings. We also connected with Krupa yesterday to line up additional testing.

Do you plan on surveying users at the end of the study? 

Not during the Beta test. More in depth surveying and testing will occur in a small Release experiment at a later date based on the outcome of the Beta study.

Comment 1

[Nate Weiner]

Pref Branch: User        //// VERY IMPORTANT

Recipe Slug: 

pref-flip-activity-stream-59-beta-pocket-sponsored-stories-bug-1432989

Comment 2

[Liz Henry (:lizzard) (relman/hg->git project)]

Can you describe or link to the test plan for this study (Krupa?)
And, who should be the assignee -- who is actually implementing this?

Thanks.

Comment 3

[anhtuanho]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #2)
> Can you describe or link to the test plan for this study (Krupa?)
> And, who should be the assignee -- who is actually implementing this?
> 
> Thanks.

if this's a question for me , i think not me , maybe i know them , a  name's style vietnamese

Comment 4

[Matt Grimes [:Matt_G]]

I've reviewed the PHD and sign off on science review for this experiment. Adding Nick for sign off due to the nature of the study.

Comment 5

[Nick Nguyen [:osunick]]

Approved.

Comment 6

[krupa raj [:krupa--use this to needinfo]]

Here is the testing doc which was used to verify this study: https://docs.google.com/document/d/1E3TXvw1Usl2oLg4uLg9bgkt028McLg7QYuy5tFpPE7M/edit

The testing was done by Peter DeHaan and Ciprian Muresan.

The only concern which QA highlighted in the Go/No-Go meeting was that the Sponsored Content (much like the rest of Activity Stream) will show up in private browsing mode if the user navigates to about:newtab or clicks on the home icon. This is being tracked at https://bugzilla.mozilla.org/show_bug.cgi?id=1433559

Since product owners are okay with the risk of shipping with that bug, QA signs off on shipping this study.

May The Force Be With You.

Comment 7

[Liz Henry (:lizzard) (relman/hg->git project)]

Sounds like we have everything in place now. Please go ahead and ship to beta users.

Comment 8

[Robert (rrayborn, he/him)]

This experiment is now live.  It as will auto-unenroll users on Feb 13th 2018 unless we are told that we need to run longer.

Comment 9

[Liz Henry (:lizzard) (relman/hg->git project)]

Rob, or Nate, can we close this bug? Is there anything left to do here?

Comment 10

[Matt Grimes [:Matt_G]]

I'd say we can close it at this point. The study has ended and all users have unenrolled.

Comment 11

[Nate Weiner]

Can be closed. Thanks!