1555817 - [Shield] Pref-Flip Study: activity-stream-pktnewtab-1555451-release67syndicatedboost

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Experimenter Bot]

activity-stream-pktnewtab-1555451-release67syndicatedboost

Currently syndicated articles compete directly against organic content in the Firefox new tab. Since the current new tab feed system is purely based on click-through rates, it is possible that a syndicated article might only appear in the new tab for a very short time. Since we have decided that a syndicated article is of sufficiently high quality to pay money for we should perhaps give it a boost to get more impressions.

We will reserve the second (middle) slot in the Firefox new tab for syndicated articles, with the goal of providing a minimum number of impressions for each syndicated article. The current target is for a syndicated article to receive 3 million impressions on the new tab (an estimated number of impressions need for $100 in revenue). Note that all solutions in this document only require changes to the Pocket Feed Machine.

There are two algorithms we want to test for selecting which article to display next:

Least Impressions First (limp): In this model we would select the syndicated article with the least number of impressions and use that for our next recommendation. This should provide a better lower bound for syndicated article impressions, biasing for newer articles first.

score: In this model we select all syndicated articles with impression counts below the cutoff (3 million), and then select the one with the highest score. This is basically using the current model on syndicated only content.

Note that we will not be tracking general browser usage stats for this experiment.

More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtab-1555451-release67syndicatedboost/

Comment 1

[Ian Wesley-Smith]

[Tracking Requested - why for this release]:

Comment 2

[Experimenter Bot]

    Experiment Type: Pref Flip Experiment

    What is the preference we will be changing

browser.newtabpage.activity-stream.feeds.section.topstories.options

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment Syndicated Score 33%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=shield_octr_fx67_b-0085-15k-age-05-syn-score-explore-locales-rr","stories_referrer":"https://getpocket.com/recommendations","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","model_keys":["nmf_model_animals","nmf_model_business","nmf_model_career","nmf_model_datascience","nmf_model_design","nmf_model_education","nmf_model_entertainment","nmf_model_environment","nmf_model_fashion","nmf_model_finance","nmf_model_food","nmf_model_health","nmf_model_home","nmf_model_life","nmf_model_marketing","nmf_model_politics","nmf_model_programming","nmf_model_science","nmf_model_shopping","nmf_model_sports","nmf_model_tech","nmf_model_travel","nb_model_animals","nb_model_books","nb_model_business","nb_model_career","nb_model_datascience","nb_model_design","nb_model_economics","nb_model_education","nb_model_entertainment","nb_model_environment","nb_model_fashion","nb_model_finance","nb_model_food","nb_model_game","nb_model_health","nb_model_history","nb_model_home","nb_model_life","nb_model_marketing","nb_model_military","nb_model_philosophy","nb_model_photography","nb_model_politics","nb_model_productivity","nb_model_programming","nb_model_psychology","nb_model_science","nb_model_shopping","nb_model_society","nb_model_space","nb_model_sports","nb_model_tech","nb_model_travel","nb_model_writing"],"show_spocs":true,"personalized":true,"version":1}

Syndicated score group will always see a syndicated article in the second slot (middle slot). The syndicated article selected will be the on with the highest score (as defined by the default feed) that is currently running.
        
- Treatment Existing New Tab 34%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=default_spocs_on","stories_referrer":"https://getpocket.com/recommendations","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","model_keys":["nmf_model_animals","nmf_model_business","nmf_model_career","nmf_model_datascience","nmf_model_design","nmf_model_education","nmf_model_entertainment","nmf_model_environment","nmf_model_fashion","nmf_model_finance","nmf_model_food","nmf_model_health","nmf_model_home","nmf_model_life","nmf_model_marketing","nmf_model_politics","nmf_model_programming","nmf_model_science","nmf_model_shopping","nmf_model_sports","nmf_model_tech","nmf_model_travel","nb_model_animals","nb_model_books","nb_model_business","nb_model_career","nb_model_datascience","nb_model_design","nb_model_economics","nb_model_education","nb_model_entertainment","nb_model_environment","nb_model_fashion","nb_model_finance","nb_model_food","nb_model_game","nb_model_health","nb_model_history","nb_model_home","nb_model_life","nb_model_marketing","nb_model_military","nb_model_philosophy","nb_model_photography","nb_model_politics","nb_model_productivity","nb_model_programming","nb_model_psychology","nb_model_science","nb_model_shopping","nb_model_society","nb_model_space","nb_model_sports","nb_model_tech","nb_model_travel","nb_model_writing"],"show_spocs":true,"personalized":true,"version":1}

The control group will receive the current new tab recommendations, where syndicated articles compete directly against organic content.

Note: we actually want an even split between all branches.
        
- Treatment Syndicated LIMP 33%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","hidden":false,"provider_icon":"pocket","provider_name":"Pocket","read_more_endpoint":"https://getpocket.com/explore/trending?src=fx_new_tab","stories_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=shield_octr_fx67_b-0085-15k-age-05-syn-limp-explore-locales-rr","stories_referrer":"https://getpocket.com/recommendations","topics_endpoint":"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US","model_keys":["nmf_model_animals","nmf_model_business","nmf_model_career","nmf_model_datascience","nmf_model_design","nmf_model_education","nmf_model_entertainment","nmf_model_environment","nmf_model_fashion","nmf_model_finance","nmf_model_food","nmf_model_health","nmf_model_home","nmf_model_life","nmf_model_marketing","nmf_model_politics","nmf_model_programming","nmf_model_science","nmf_model_shopping","nmf_model_sports","nmf_model_tech","nmf_model_travel","nb_model_animals","nb_model_books","nb_model_business","nb_model_career","nb_model_datascience","nb_model_design","nb_model_economics","nb_model_education","nb_model_entertainment","nb_model_environment","nb_model_fashion","nb_model_finance","nb_model_food","nb_model_game","nb_model_health","nb_model_history","nb_model_home","nb_model_life","nb_model_marketing","nb_model_military","nb_model_philosophy","nb_model_photography","nb_model_politics","nb_model_productivity","nb_model_programming","nb_model_psychology","nb_model_science","nb_model_shopping","nb_model_society","nb_model_space","nb_model_sports","nb_model_tech","nb_model_travel","nb_model_writing"],"show_spocs":true,"personalized":true,"version":1}

Syndicated LIMP group will always see a syndicated article in the second slot (middle slot). The syndicated article selected will be the on with the least impressions that is currently running.
        

    What version and channel do you intend to ship to?

0.9% of Release Firefox 67.0

    Are there specific criteria for participants?

Note that we actually are fine with users from FF66-68
Prefs:

Studies:
We don't want users who are in discovery stream layout experiments. Specifically please exclude any users in the following experiments:
* https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh12019-release66layoutstest1/
* https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh12019-release66-confirmation-test/

Any additional filters:

    What is your intended go live date and how long will the experiment run?

Jun 24, 2019 - Jul 22, 2019 (28 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

We will be measuring the following metrics for each group:
* Impressions (syndicated/organic/all)
* Clicks (syndicated/organic/all)

Do you plan on surveying users at the end of the study? No.

    Who is the owner of the data analysis for this experiment?

Ian Wesley-Smith

    Will this experiment require uplift?

False

    QA Status of your code:

Not Started

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtab-1555451-release67syndicatedboost/

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Per Experimenter, this was ended on July 9.