Closed Bug 1633672 Opened 6 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-personalization-v2-1621018 Fx 76.0 to 77.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox76+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox76 + fixed

People

(Reporter: experimenter, Assigned: digitalnewslab)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

browser.newtabpage.activity-stream.discoverystream.personalization.version

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment variant 50%:

Value: 2

Stories are recommended using interest topics
        
- Control control 50%:

Value: 1

Stories are recommended using domain affinities
        

    What version and channel do you intend to ship to?

1% of Release Firefox 76.0 to 77.0

    Are there specific criteria for participants?

Prefs:

browser.newtabpage.enabled
OR (we want AT LEAST one of these to be default)
browser.startup.homepage

Experiments: 
Exclude users enrolled in the Hero Layout Test
https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-hero-layout-test-1624064/

Any additional filters:

AND (we want ALL of the below to be default)
browser.newtabpage.activity-stream.showSearch
AND
browser.newtabpage.activity-stream.showSponsored
AND
browser.newtabpage.activity-stream.feeds.topsites
AND
browser.newtabpage.activity-stream.feeds.section.topstories
AND
browser.newtabpage.activity-stream.feeds.section.highlights
AND
browser.newtabpage.activity-stream.discoverystream.config
AND
browser.newtabpage.activity-stream.discoverystream.personalization.modelKeys
Countries: United States of America (US) 

Locales: English (US) (en-US) 

    What is your intended go live date and how long will the experiment run?

May 12, 2020 - Jun 09, 2020 (28 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

The main effect we're looking for is whether users will engage more with sponsored content items relevant to the general topics they are interested in.

Our main success metrics are:  
1. Higher CTR on personalized SPOCS versus non-personalized SPOCS. 
2. Lower dismiss rate on personalized SPOCS versus non-personalized. 

Do you plan on surveying users at the end of the delivery? Yes.
Strategy and Insights can help create surveys if needed

    Who is the owner of the data analysis for this experiment?

jkoren@getpocket.com

    Will this experiment require uplift?

False

    QA Status of your code:

In Progress

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-personalization-v2-1621018/

activity-stream-pocket-new-tab-personalization-v2-1621018

Topic-based Sponsored Content tagging for better relevance.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-352
More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-personalization-v2-1621018/

[Tracking Requested - why for this release]:

tracking-firefox76: --- → ?

SPOC Targeting V2 Experiment
Firefox Release 76

We have finished testing the [PI-532] - SPOC Targeting V2 experiment.

Quality status: GREEN - SHIP IT

Why is this experiment GREEN?

    • The new issues found during testing are general issues (not related to this experiment) and do not impact users in a harmful way.

Testing Summary:

Tested Platforms:

  • Windows 10 x64
  • Mac OS 10.15;
  • Linux Ubuntu 18.04 x64;

Tested browser versions:

  • Firefox RC 76 (en-US)

Regards,
Valentina Virlics
Ecosystem QA

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: activity-stream-pocket-new-tab-personalization-v2-1621018 → [Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-personalization-v2-1621018 Fx 76.0 to 77.0 Release
Start Date: 2020-05-12 End Date: 2020-06-09
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
status-firefox76: affected → fixed
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.