Closed Bug 1582844 Opened 6 years ago Closed 6 years ago

[Experiment] Pref-Flip Experiment: activity-stream-pktnewtabH22019-70-personalization-test Fx 69.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: experimenter, Assigned: digitalnewslab)

References

(
URL
)

Details

User Story

Experiment Type: Pref Flip Experiment

    What is the preference we will be changing

browser.newtabpage.activity-stream.discoverystream.config

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment Personalization ON 50%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","collapsible":true,"enabled":true,"show_spocs":true,"hardcoded_layout":true,"personalized":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic", "foo":"ba"}

New Tab Experience with Personalization Turned ON
        
- Treatment Personalization OFF 50%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","collapsible":true,"enabled":true,"show_spocs":true,"hardcoded_layout":true,"personalized":false,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic", "foo":"ba"}

New Tab Experience with Personalization Turned OFF
        

    What version and channel do you intend to ship to?

1% of Release Firefox 69.0

    Are there specific criteria for participants?

Prefs:

browser.newtabpage.enable
OR (we want AT LEAST one of these to be default)
browser.startup.homepage

Enroll Profiles that are on: 
• FF69

AND (we want ALL of the below to be default)

browser.newtabpage.activity-stream.showSearch
AND
browser.newtabpage.activity-stream.feeds.topsites
AND
browser.newtabpage.activity-stream.feeds.section.topstories
AND
browser.newtabpage.activity-stream.feeds.section.highlights

Please exclude all subjects that were enrolled (at any point) in the following experiment:
https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-new-tab-release69-layout-holdback-1573312/
Countries: United States of America (US) 

Locales: English (US) (en-US) 

    What is your intended go live date and how long will the experiment run?

Oct 01, 2019 - Nov 05, 2019 (35 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

We would like to improve engagement with Sponsored Content. In order to show users content relevant to them we would like to measure the impact of personalization on engagement with SPOCS. 

Do you plan on surveying users at the end of the experiment? No.

    Who is the owner of the data analysis for this experiment?

kirill@getpocket.com

    Will this experiment require uplift?

False

    QA Status of your code:

Signed off as GREEN, for more information see: https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh22019-70-personalization-test/#comment455

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh22019-70-personalization-test/

activity-stream-pktnewtabH22019-70-personalization-test

Pocket New Tab study to measure the impact of Personalization on SPOC performance.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh22019-70-personalization-test/

We have finished testing the activity-stream-pktnewtabH22019-70-personalization-test
experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • We have not found any new issues during testing the experiment and the known issues do not affect the user in a harmful way.

Testing Summary:

  • TestRail link: Link.

Tested Platforms:

  • Windows 10
  • macOS 10.14
  • Linux Mint 19

Tested Firefox versions:

  • Firefox Release 69.0.1

Regards,
Ciprian Muresan

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: activity-stream-pktnewtabH22019-70-personalization-test → [Experiment] Pref-Flip Experiment: activity-stream-pktnewtabH22019-70-personalization-test Fx 69.0 Release
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.