Open Bug 1458310 Opened 6 years ago Updated 2 years ago

[Shield] Pref Flip Study: New Tab Spocs Optimization Test

Categories

(Shield :: Shield Study, defect)

Product:
Shield
Component:
Shield Study
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

People

(Reporter: nate+bugzilla, Unassigned)

References

Details

Basic description of experiment: 

Pocket sponsored stories are integrated into the new tab page via Activity Stream. This test creates branches that allow for testing of different fill rates, frequency caps, personalization algorithms for optimization of Spoc performance

More details about the test and why we’re doing it is on Mozilla’s blog here: 
https://blog.mozilla.org/futurereleases/2018/04/30/a-privacy-conscious-approach-to-sponsored-content/

What is the preference we will be changing? 

browser.newtabpage.activity-stream.feeds.section.topstories.options

What are the branches of the study and what values should each branch be set to? 

Control:
{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=shield_optimization_fx60_spocs_off","stories_referrer":"https://getpocket.com/recommendations","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":false,"personalized":true}

Fill30:
{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=shield_optimization_fx60_spocs_fill30","stories_referrer":"https://getpocket.com/recommendations","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":true,"personalized":true}

Fill50:
{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=shield_optimization_fx60_spocs_fill50","stories_referrer":"https://getpocket.com/recommendations","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":true,"personalized":true}

Title1:
{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=shield_optimization_fx60_spocs_title1","stories_referrer":"https://getpocket.com/recommendations","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":true,"personalized":true}

Title2:
{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=shield_optimization_fx60_spocs_title2","stories_referrer":"https://getpocket.com/recommendations","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":true,"personalized":true}

Title3:
{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=shield_optimization_fx60_spocs_title3","stories_referrer":"https://getpocket.com/recommendations","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":true,"personalized":true}

Personalization1:
{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=shield_optimization_fx60_spocs_personalization1","stories_referrer":"https://getpocket.com/recommendations","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":true,"personalized":true}

Personalization2:
{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=shield_optimization_fx60_spocs_personalization2","stories_referrer":"https://getpocket.com/recommendations","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":true,"personalized":true}

Cap1:
{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=shield_optimization_fx60_spocs_cap1","stories_referrer":"https://getpocket.com/recommendations","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":true,"personalized":true}

Cap2:
{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=shield_optimization_fx60_spocs_cap2","stories_referrer":"https://getpocket.com/recommendations","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":true,"personalized":true}


What percentage of users do you want in each branch? 

The study should be for 20% of US/en-US, excluding users in the 2% holdback study

Within that 20%, the variants should be divided as follows:

Control: 5%
Fill30: 5%
Fill50: 5%
Title1: 15%
Title2: 15%
Title3: 15%
Personalization1: 10%
Personalization2: 10%
Cap1: 10%
Cap2: 10%

What Channels and locales do you intend to ship to? 

Channel: All Channels >= 60
Locale: en-US
Geo: US

What is your intended go live date and how long will the study run? 

Start: Monday May 7th 2018
End: Friday, August 10th

Are there specific criteria for participants? 
No

What is the main effect you are looking for and what data will you use to make these decisions? 

Firefox retention and click through rates of sponsored stories.

Who is the owner of the data analysis for this study? 

Kenny Long

Will this experiment require uplift? 

No

QA Status of your code: 

QA Request has been submitted. Additionally, this codebase has been in Firefox since Firefox 58 and has been subject to Beta and Release tests, as well as QA'd for 60 through the holdback test.

Do you plan on surveying users at the end of the study? 

No.

Additional Materials:

Holdback Test: https://bugzilla.mozilla.org/show_bug.cgi?id=1456585
Pref Branch: User        //// VERY IMPORTANT

Recipe Slug: 

pref-flip-activity-stream-60-release-pocket-spocs-optimization-1458310
No longer depends on: 1459221
Can I get a science signoff and relman signoff, please? Also need QA signoff. Krupa, is that you? Thanks!
Flags: needinfo?(kraj)
Flags: needinfo?(jcristau)
Flags: needinfo?(isegall)
Marnie - it looks like got QA signoff this morning from Marius (QA) this morning.  

Ilana  - are you able to sign off from science?

Julien - do you need anything further from your end?
Flags: needinfo?(kraj)
Also adding back in needsinfo from Krupa until we get confirmation that the email this morning from our QA contact (Marius) is sufficient to remove it.
Flags: needinfo?(kraj)
Science: R+
Flags: needinfo?(isegall)
Adding Dave for additional sign off based on the risk matrix.
Flags: needinfo?(dcamp)
QA notes:
------------------------------------------------------------------------
Pocket Spocs Optimization Test Shield Study experiment
Targeted: All channels, version >= 60

We have finalized testing the "Pocket Spocs Optimization Test" Shield Study experiment.

QA’s recommendation: YELLOW - ship it conditionally

Reasoning: The shield study overall state is good and the remaining issues are not major. The only concern we have is issue #1458906 which has an active uplift request. If the fix won't be uplifted in Beta 60, the issue could affect the enrolled population from the next Firefox Release, since the study targets Fx60 and Fx61.

Testing Summary:
- Verified that the "Sponsored content" stories (“SPOCS”) only show in expected cases/variants.
- Verified that the users are able to disable sponsored stories.
- Verified that the "Sponsored content" loads (in general)
- Verified that the "Sponsored content" frequency caps perform as described
- Verified that the "Sponsored content" is only shown in the US + en-US

Tested Firefox versions:
- Firefox Beta build v60;
- Firefox Nightly build v61.0a1;

------------------------------------------------------------------------

The only concern was https://bugzilla.mozilla.org/show_bug.cgi?id=1458906 but as per Nate, this was deemed as not-blocking for this launch.

So, ship it.
Flags: needinfo?(kraj)
Julien (FYI) and I both reviewed the blocking issue Krupa mentioned and I was told this would impact ~0.01% of the release population based on which it was deemed non-blocking. With that taken care of and the QA sign-off, we are ready to let this go live.

Matt_G mentioned a slow rollout on this one, given the ~20% en-US release population target. +1 to that plan.
Bug 1458906 was added to release notes for 60 as a known issue.

My only (slight) worry is the timing vs the 60 release, but based on the QA notes it sounds like we're fine, so happy to sign off for relman.

One question: does "all channels" in comment 0 include esr, or only nightly/DevEd/beta/release?
Flags: needinfo?(jcristau)
Approved.
Flags: needinfo?(dcamp)
> One question: does "all channels" in comment 0 include esr, or only
> nightly/DevEd/beta/release?

The PHD document had the channels/versions section filled out as all versions >=60, so I presume this applies to release, beta and Nightly channel. If the study owner can confirm it, that would be great.
:ritu You are correct, the target is release, beta and Nightly channels (cc :jcristau)
After a quick conversation with the pocket team about this we re-scoped to focus on Release and make Relman's life easier.

We are live.  Let us know if you need more sample.
(In reply to krupa raj [:krupa--use this to needinfo] from comment #13)
> :ritu 
> is https://bugzilla.mozilla.org/show_bug.cgi?id=1458906#c12 a concern?

Nope. As I mentioned in comment 8, this impacts a very small segment of the population + not-so-severe (like a hang/crash/perf impact).
Flags: needinfo?(rkothari)
Depends on: 1477380
NI'ing Mythmon.  A bug in User branch prefs has been discovered that will need fixed before we go live with this.  Luckily this is Nightly so we won't be waiting long; unluckily this bug almost certainly opted a large number of users out of our long term holdback tests.  Expect more info soon.
Flags: needinfo?(mcooper)
(In reply to Rob from comment #16)
> NI'ing Mythmon.  A bug in User branch prefs has been discovered that will
> need fixed before we go live with this.  Luckily this is Nightly so we won't
> be waiting long; unluckily this bug almost certainly opted a large number of
> users out of our long term holdback tests.  Expect more info soon.

Rrayborn: this study is live. Did you mean to add your comment to this pending Nightly study: https://bugzilla.mozilla.org/show_bug.cgi?id=1477083 ?  And am I reading correctly that the bug you referenced blocks that Nightly study from going live until fixed?
The bug mentioned in comment 16 was bug 1477380. It has been fixed on Nightly, as of buildid 20180725103029. It has also been uplifted to beta, and will be included in the next beta build (I'm not sure when that will be).
Flags: needinfo?(mcooper)
Per conversation with :marnie a few months ago the end date for this study should be updated to 5/18/19.
[Experiment]Pocket Spocs Optimization Test status has been changed to: Complete
            url:https://experimenter.services.mozilla.com/experiments/pref-flip-activity-stream-60-release-pocket-spocs-optimization-1458310/
You need to log in before you can comment on or make changes to this bug.