Open Bug 1456585 Opened 2 years ago Updated 2 years ago

[Shield] Pref Flip Study: New Tab Spocs Holdback Test

Categories

(Shield :: Shield Study, enhancement)

enhancement
Not set

Tracking

(Not tracked)

People

(Reporter: nate, Unassigned)

Details

Basic description of experiment: 

Pocket recommendations are integrated into the new tab page via Activity Stream. For users in the test groups, they would be eligible to see a sponsored story periodically in the 3rd position of the recommendations section. This experiment enables the ability for these stories to be shown as well as an additional option inside of the New Tab options pane that allows users to turn off sponsored stories if they choose to. For users in the control group, they will not see any sponsored stories.

More details about the test and why we’re doing it is on Mozilla’s blog here: 
https://blog.mozilla.org/futurereleases/2018/01/24/update-on-content-recommendations-in-firefox-new-tab-page/

What is the preference we will be changing? 

browser.newtabpage.activity-stream.feeds.section.topstories.options

What are the branches of the study and what values should each branch be set to? 

Control: 

{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=shield_holdback_fx60_spocs_off","stories_referrer":"https://getpocket.com/recommendations","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":false,"personalized":true}

Spocs Variant:

{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=shield_holdback_fx60_spocs_on","stories_referrer":"https://getpocket.com/recommendations","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":true,"personalized":true}

What percentage of users do you want in each branch? 

The total should be only be deployed to 2% of Release population. Within that 2%, it should be divided into 2 branches:

Control: 50%
Spocs: 50%

What Channels and locales do you intend to ship to? 

Channel: All Channels >= 60
Locale: en-US
Geo: US

What is your intended go live date and how long will the study run? 

The study will run indefinitely (pending periodic check-in on control to determine no divergence between holdback and spocs):
Start: Monday April 30th 2018
End: N/A

Are there specific criteria for participants? 
No

What is the main effect you are looking for and what data will you use to make these decisions? 

The main measurement is Firefox retention. Our goal is to see zero to minimal impact on retention. Additionally, we’ll be looking at the performance of the various sponsored stories (Click-through rates, dismiss rates, etc).

Who is the owner of the data analysis for this study? 

Kenny Long

Will this experiment require uplift? 

No

QA Status of your code: 

QA Request has been submitted. Additionally, this codebase has been in Firefox since Firefox 58 and has been subject to Beta and Release tests.

Do you plan on surveying users at the end of the study? 

No.

Additional Materials:

Sponsored Study in Firefox 58: https://bugzilla.mozilla.org/show_bug.cgi?id=1435822
Pref Branch: User        //// VERY IMPORTANT

Recipe Slug: 

pref-flip-activity-stream-60-release-pocket-spocs-holdback-1456585
We have finished testing the "New Tab Sponsored Content Holdback Test" Shield Study experiment.

Reasoning: We have found no issues during testing and as a result we consider that the study can be released to end users.

QA’s recommendation: GREEN

Testing Summary:
- Verified that Sponsored content does not appear and also that the "Sponsored Stories" checkbox does not appear in Control.
- Verified that Sponsored content appears and also that the "Sponsored Stories" checkbox is displayed in the Spocs variant.
- Verified that Sponsored content no longer appears after the "Sponsored Stories" checkbox is unchecked.
- Verified that Sponsored content reappears after the "Sponsored Stories" checkbox is checked.
- Verified that the Telemetry ping containing << "event":"topstories.domain.affinity.calculation.ms" >> appears.

Tested Platforms:
- Windows 10 x64;
- Mac OS 10.13.3;
- Ubuntu 16.04 x32;

Tested Firefox versions:
- Firefox Beta build v60.0b15;
- Firefox Nightly build v61.0a1;
No concerns from release management for a rollout to 2% of en-US 60+.
Science review: R+
The tl;dr on this study (for those catching up):

Sponsored Content in Pocket Recommendations is being rolled out as the default experience in Fx 60 new tab, and this study will measure the impact on long-term retention of this by turning sponsored content OFF for 1% of users.
Thanks for the recap Kenny. Since this has partner implications I'm flagging DCamp for a final signature.
Flags: needinfo?(dcamp)
Approved.
Flags: needinfo?(dcamp)
The study is live.
You need to log in before you can comment on or make changes to this bug.