1425490 - [Shield] Pref Flip Study: Activity Stream 58 Beta Pocket Personalization

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Jared Lockhart [:jlockhart]]

Basic description of experiment: 

Pocket is integrated into the new tab page via Activity Stream. All users presently receive the same set of stories in the same order.  By analyzing a users history locally, we can determine a set of domains they visit frequently, and use that data to infer which stories would be more relevant to that user.  All of this processing will happen client side.  This mechanism is default enabled in 58 Beta, however the server side component which actually presents stories that can be matched to clients personalization parameters is not.  This experiment will enable that server side data integration to enable personalization of stories for the variant users.

What is the preference we will be changing? 

browser.newtabpage.activity-stream.feeds.section.topstories.options

What are the branches of the study and what values should each branch be set to? 

Control: 

{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_header":"pocket_feedback_header","provider_description":"pocket_description","provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=2&consumer_key=$apiKey&locale_lang=en-US","stories_referrer":"http://getpocket.com/recommendations","info_link":"https://www.mozilla.org/privacy/firefox/#pocketstories","disclaimer_link":"https://getpocket.cdn.mozilla.net/firefox/new_tab_learn_more","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":false,"personalized":true}


Personalized:

{"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_header":"pocket_feedback_header","provider_description":"pocket_description","provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=2&consumer_key=$apiKey&locale_lang=en-US&feed_variant=personalized_v1","stories_referrer":"http://getpocket.com/recommendations","info_link":"https://www.mozilla.org/privacy/firefox/#pocketstories","disclaimer_link":"https://getpocket.cdn.mozilla.net/firefox/new_tab_learn_more","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","show_spocs":false,"personalized":true}

What percentage of users do you want in each branch? 

100% of 58 Beta with 1 Control : 1 Personalized

What Channels and locales do you intend to ship to? 

Channel: 58 Beta
Locale: en-US
Geo: US

What is your intended go live date and how long will the study run? 

Start: Fri Dec 15th 2017
End: Jan 23rd 2018

Are there specific criteria for participants? 

No

What is the main effect you are looking for and what data will you use to make these decisions? 

By personalizing stories we should theoretically see an increase in engagement with Pocket stories for the 'personalized' variant.  This can be evaluated by looking at the click through rates on Pocket stories using the Ping Centre dataset.  We hope to see no regressions in any other engagement or performance metrics, either in Ping Centre or Firefox Telemetry.

Who is the owner of the data analysis for this study? 

Marina Samuel 

Will this experiment require uplift? 

No

QA Status of your code: 

This code has been tested and QAed in 58 Nightly, multiple studies have previously been run against it, and it is default enabled in 58 Beta.  This change only alters a URL parameter when data is fetched from the server.

Do you plan on surveying users at the end of the study? 

No

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:

This study builds on the following three previous studies:

https://bugzilla.mozilla.org/show_bug.cgi?id=1400890 (contains two studies)
https://bugzilla.mozilla.org/show_bug.cgi?id=1410483

Comment 1

[Jared Lockhart [:jlockhart]]

Pref Branch: User        //// VERY IMPORTANT

Recipe Slug: 

pref-flip-activity-stream-58-beta-pocket-personalization-bug-1425490

Comment 2

[Michael Cooper [:mythmon]]

(In reply to Jared Kerim [:jkerim] from comment #0)
> What is the preference we will be changing? 
> 
> browser.newtabpage.activity-stream.feeds.section.topstories.options

This pref contains JSON, which is a big red flag for not being suitable to use in a Shield study. Shield assumes that nothing else will change the preference's value during a study, and if it does change, that was a user choice, which causes the study to end, with no option to restart it.

This is one of the major problems that past Activity Stream studies have had. The general recommendation for Shield studies is to have a preference that contains a single, simple value that doesn't change as a matter of general operation. Several fields in the example payloads seem like they store state that will or could change over time.

I recommend not running this study as-is.

Comment 3

[Nate Weiner]

(In reply to Mike Cooper [:mythmon] from comment #2)

> This pref contains JSON, which is a big red flag for not being suitable to
> use in a Shield study. Shield assumes that nothing else will change the
> preference's value during a study, and if it does change, that was a user
> choice, which causes the study to end, with no option to restart it.

Hey Mike - This pref will not change based on general operation or user input. The only modifications to this pref come from shield studies or new releases where we want to change behavior (based on test results). We were purposeful about how we used that pref for this specific reason.

Hopefully that addresses the concern as you heard from Denelle during the Plenary: This is an important study to get launched before the break starts.

If moving forward the preference is not to use JSON for these studies, we can work to get onboard with that in future releases, but doing so would prevent us from releasing this until 59.

Comment 4

[Michael Cooper [:mythmon]]

If you're confident the preference won't change, then there isn't a real problem here.

Comment 5

[Jared Lockhart [:jlockhart]]

Just as a followup to :mythmon's concern, it is true that in the past we have seen issues surrounding Activity Stream overwriting a pref and causing the Shield system addon to detect a pref change and thus unenroll users.  

When the Activity Stream system addon starts up on browser init, it overwrites all of its prefs with its latest default values (regarldess of whether those defaults have changed), but only so long as that pref is in its 'default' state.  To prevent this from causing Shield unenrollments, we now launch all Activity Stream studies using the 'User' pref branch, and have seen that this does prevent the unenrollment issues.

Ideally we will find a better solution in the future and change the way that Activity Stream overwrites prefs at startup, but in the mean time I believe this approach should let us continue to run studies.

Comment 6

[Matt Grimes [:Matt_G]]

If mythmon is not concerned from a technical standpoint I think it's fine. Giving my stamp of approval here. Jared and I can discuss timing for shipping this since we couldn't do it over the holiday break.

Comment 7

[Gerry Chang [:gchang]]

Hi Matt,
We are good to go to collect data.

Comment 8

[Matt Grimes [:Matt_G]]

This study is now live on Beta for Geo US. The study will end Jan 23rd.

Comment 9

[Matt Grimes [:Matt_G]]

@jkerim Recap and close this bug for me?

Comment 10

[Jared Lockhart [:jlockhart]]

The dashboard for this experiment can be found here:

https://sql.telemetry.mozilla.org/dashboard/pocket-experiment-beta-personalization


The data shows that there was a neutral effect on all firefox engagement measurements that we normally monitor.  This is the ideal outcome of this experiment, it shows us that the implementation of this feature does not impact the user experience enough to cause a drop in usage, which is what we hoped to find.  This does not measure any kind of 'lift' in the actual Pocket click through rate, which would require a separate analysis from the Pocket team.  Closing this bug.