Closed Bug 1654914 Opened 6 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-story-spocs-in-uk-experiment Fx 79.0 to 81.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox79+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox79 + fixed

People

(Reporter: experimenter, Assigned: digitalnewslab)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

browser.newtabpage.activity-stream.discoverystream.region-spocs-config

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment variant 50%:

Value: US,CA,GB

spocs turned on in GB
        
- Control control 50%:

Value: US,CA

spocs turned off in GB
        

    What version and channel do you intend to ship to?

50% of Release Firefox 79.0 to 81.0

    Are there specific criteria for participants?

Preset Common Pocket Filter 1
Countries: United Kingdom (GB) 

Locales: English (British) (en-GB) English (Canadian) (en-CA) English (US) (en-US) 

    What is your intended go live date and how long will the experiment run?

Aug 25, 2020 - Sep 29, 2020 (35 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

The main effect we're looking for is whether users will engage with Sponsored Content without regressing engagement with Pocket stories, Firefox browser usage, and Search.


Estimated metric impact:

-Sponsored Content CTR of .06% (based on US engagement with Sponsored Content)

-Story CTR of .45% (based on current UK engagement with the assumption that Story CTR doesn't change significantly since users are much more likely to click Stories than Sponsored Content)

-Story Clicks per User relative change of -2% (based on testing Sponsored Content in DE)

-% of Users that Clicked Stories relative change of -8% (based on testing Sponsored Content in DE)

-% of Users with Pocket Disabled relative change of +15% (based on testing Sponsored Content in DE)


Do you plan on surveying users at the end of the delivery? No. Strategy and Insights can help create surveys if needed

    Who is the owner of the data analysis for this experiment?

kirill@getpocket.com

    Will this experiment require uplift?

False

    QA Status of your code:

QA Status: GREEN. For more information see https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-story-spocs-in-uk-experiment/#comment1052.

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-story-spocs-in-uk-experiment/

activity-stream-pocket-new-tab-story-spocs-in-uk-experiment

Study to measure engagement with Sponsored Content in the UK.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-352
More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-story-spocs-in-uk-experiment/

[Tracking Requested - why for this release]:

tracking-firefox79: --- → ?

New Tab SPOCS in UK
Firefox Release 79

We have finished testing the New Tab SPOCS in UK experiment.

QA’s recommendation: GREEN - SHIP IT

Why is this experiment green?

  • We haven’t found any issues during testing.

Testing Summary:

  • TestRail: Link
  • Verified that SPOCs appeared with the US and GB overrides (browser.newtabpage.activity-stream.discoverystream.spocs-endpoint set to https://spocs.getpocket.com/spocs?country=US&region=NY for US and https://spocs.getpocket.com/spocs?country=GB&region=UK for GB).

Tested Platforms:

  • Windows 10 x64
  • macOS 10.15
  • Ubuntu 20.04

Tested Firefox versions:

  • Firefox RC 79.0
  • Firefox Beta 80.0b1 (update test only)

Regards,
Ciprian Muresan
Ecosystem QA Team

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: activity-stream-pocket-new-tab-story-spocs-in-uk-experiment → [Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-story-spocs-in-uk-experiment Fx 79.0 to 81.0 Release
Start Date: 2020-08-25 End Date: 2020-09-29
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
status-firefox79: affected → fixed
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.