Closed Bug 1634769 Opened 4 years ago Closed 4 years ago

[Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-hero-layout-smoke-test-1624064 Fx 77.0 to 78.0 Beta

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox77+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox77 + fixed

People

(Reporter: experimenter, Assigned: digitalnewslab)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

browser.newtabpage.activity-stream.discoverystream.config

    What are the branches of the experiment and what values should
    each branch be set to?

- Control 3 Columns 7 Rows 50%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","collapsible":true,"enabled":true,"show_spocs":true,"hardcoded_layout":true,"personalized":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic","unused_key":"unused_2"}

Default New Tab layout with 3 Columns and 7 Rows of tiles.
        
- Treatment Hero Layout 50%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","collapsible":true,"enabled":true,"show_spocs":true,"hardcoded_layout":false,"personalized":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=hero-layout","unused_key":"unused_2"}

New Hero layout with hero element above 6 rows of tiles.
        

    What version and channel do you intend to ship to?

1% of Beta Firefox 77.0 to 78.0

    Are there specific criteria for participants?

browser.newtabpage.enable
OR (we want AT LEAST one of these to be default)
browser.startup.homepage


AND (we want ALL of the below to be default)
browser.newtabpage.activity-stream.showSearch
AND
browser.newtabpage.activity-stream.showSponsored
AND
browser.newtabpage.activity-stream.feeds.topsites
AND
browser.newtabpage.activity-stream.feeds.section.topstories
AND
Browser.newtabpage.activity-stream.feeds.section.highlights
AND
browser.newtabpage.activity-stream.discoverystream.config
AND
browser.newtabpage.activity-stream.discoverystream.personalization.modelKeys


Experiments: None
Countries: United States of America (US) 

Locales: English (US) (en-US) 

    What is your intended go live date and how long will the experiment run?

May 12, 2020 - May 26, 2020 (14 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

If we see higher engagement with Hero Layout we will make it the default New Tab layout. Our primary success metrics will be (1) the % of each variant that engaged with Pocket recs at least once during the test and (2) the click rate on the 1st position Pocket rec. This latter metric is important, as it allows us to understand whether showing Syndicated content in the Hero position can send more traffic there.


Do you plan on surveying users at the end of the delivery?  No.
Strategy and Insights can help create surveys if needed

    Who is the owner of the data analysis for this experiment?

kirill@getpocket.com

    Will this experiment require uplift?

False

    QA Status of your code:

QA Status : GREEN. 
For more info see https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-hero-layout-smoke-test-1624064/#comment841 .

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-hero-layout-smoke-test-1624064/

activity-stream-pocket-new-tab-hero-layout-smoke-test-1624064

Pocket New Tab study comparing engagement with stories and SPOCs between hero layout and Control.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-352
More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-hero-layout-smoke-test-1624064/

[Tracking Requested - why for this release]:

tracking-firefox77: --- → ?
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: activity-stream-pocket-new-tab-hero-layout-smoke-test-1624064 → [Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-hero-layout-smoke-test-1624064 Fx 77.0 to 78.0 Beta

New Tab Layouts - Beta 77 Smoke test Experiment
Firefox Beta 77

  • We have finished testing the [PI-531] - New Tab Layouts - Beta 77 Smoke test Experiment.

Quality status: GREEN - SHIP IT

Why is this experiment GREEN?

  • The new issues found during testing are minor issues (with severity S4) and do not impact users in a harmful way.

Testing Summary:

Tested Platforms:

  • Windows 10 x64;
  • Mac OS 10.15;
  • Linux Ubuntu 18.04 x64;

Tested browser versions:

  • Firefox Beta 77 (en-US);

Regards,
Valentina Virlics
Ecosystem QA

User Story: (updated)
User Story: (updated)
Start Date: 2020-05-12 End Date: 2020-05-26
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.