Closed Bug 1646076 Opened 5 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-popular-topics-smoke-test Fx 78.0 to 79.0 Beta

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox78+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox78 + fixed

People

(Reporter: experimenter, Assigned: digitalnewslab)

References

(
URL
)

Details

(Whiteboard: [no-nag])

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

browser.newtabpage.activity-stream.discoverystream.config

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment Variant 50%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","collapsible":true,"enabled":true,"show_spocs":true,"hardcoded_layout":false,"personalized":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=topics-improved-layout", "unused_key":"unused_3"}

Seeing new popular topics above the fold
        
- Control Control 50%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","collapsible":true,"enabled":true,"show_spocs":true,"hardcoded_layout":true,"personalized":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic", "unused_key":"unused_3"}

Seeing old popular topics below the fold
        

    What version and channel do you intend to ship to?

1% of Beta Firefox 78.0 to 79.0

    Are there specific criteria for participants?

Prefs:

browser.newtabpage.enabled
OR (we want AT LEAST one of these to be default)
browser.startup.homepage

AND (we want ALL of the below to be default)

browser.newtabpage.activity-stream.showSearch
AND
browser.newtabpage.activity-stream.feeds.topsites
AND
browser.newtabpage.activity-stream.feeds.section.topstories
AND
browser.newtabpage.activity-stream.feeds.section.highlights

Experiments: None
Countries: United States of America (US) 

Locales: English (US) (en-US) 

    What is your intended go live date and how long will the experiment run?

Jun 23, 2020 - Jul 07, 2020 (14 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

The main effect we're looking for is whether Popular Topics positioned above the fold see a higher volume of clicks per user than when they are positioned below the fold. We will also be checking for any corresponding decline in engagement with stories and SPOCS.

Do you plan on surveying users at the end of the delivery?  No.
Strategy and Insights can help create surveys if needed

    Who is the owner of the data analysis for this experiment?

kirill@getpocket.com

    Will this experiment require uplift?

False

    QA Status of your code:

QA Status: GREEN. For more information, see: https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-popular-topics-smoke-test/#comment939

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-popular-topics-smoke-test/

activity-stream-pocket-new-tab-popular-topics-smoke-test

New Tab Popular Topics Smoke Test

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-352
More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-popular-topics-smoke-test/

New Tab Layout Study - Popular Topics - Firefox Beta 78 (6/23)

Firefox Beta 78

We have finished testing the [PI-668] New Tab Layout Study - Popular Topics - Firefox Beta 78 (6/23) experiment.

QA’s recommendation: GREEN-SHIP IT

Why is this experiment green?

  • During testing, we have uncovered 3 new issues. From the encountered issues, one is not specific to the experiment in the test and the other two are not considered of high impact for the user experience.

Testing Summary:

Tested Platforms:

  • Windows 10 x64
  • macOS 10.15
  • Ubuntu 18.04 x64

Tested Firefox versions:

  • Firefox Beta 78.0b8

[Tracking Requested - why for this release]:
78 Beta smoketest experiment to make sure all is good for an experiment we want to launch in Release 78.

tracking-firefox78: --- → ?
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: activity-stream-pocket-new-tab-popular-topics-smoke-test → [Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-popular-topics-smoke-test Fx 78.0 to 79.0 Beta
status-firefox78: --- → affected
tracking-firefox78: ? → +
User Story: (updated)
Whiteboard: [no-nag]
Start Date: 2020-06-23 End Date: 2020-07-07
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.