[Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-popular-topics-release-test-v3 Fx 80.0 to 81.0 Release
Categories
(Shield :: Shield Study, task, P3)
Tracking
(firefox80+ fixed)
People
(Reporter: experimenter, Assigned: kdemtchouk)
References
()
Details
User Story
Delivery Type: Pref Flip Experiment What is the preference we will be changing browser.newtabpage.activity-stream.discoverystream.config What are the branches of the experiment and what values should each branch be set to? - Treatment Variant 50%: Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","collapsible":true,"enabled":true,"show_spocs":true,"hardcoded_layout":false,"personalized":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=topics-improved-layout", "unused_key":"unused_3"} Seeing new popular topics above the fold - Control Control 50%: Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","collapsible":true,"enabled":true,"show_spocs":true,"hardcoded_layout":true,"personalized":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic", "unused_key":"unused_3"} Seeing old popular topics below the fold What version and channel do you intend to ship to? 1% of Release Firefox 80.0 to 81.0 Are there specific criteria for participants? Prefs: browser.newtabpage.enabled OR (we want AT LEAST one of these to be default) browser.startup.homepage AND (we want ALL of the below to be default) browser.newtabpage.activity-stream.showSearch AND browser.newtabpage.activity-stream.feeds.topsites AND browser.newtabpage.activity-stream.feeds.section.topstories AND browser.newtabpage.activity-stream.feeds.section.highlights Experiments: None Countries: United States of America (US) Locales: English (US) (en-US) What is your intended go live date and how long will the experiment run? Sep 01, 2020 - Oct 06, 2020 (35 days) What is the main effect you are looking for and what data will you use to make these decisions? The main effect we're looking for is whether Popular Topics positioned above the fold see a higher volume of clicks per user than when they are positioned below the fold. We will also be checking for any corresponding decline in engagement with stories and SPOCS. Do you plan on surveying users at the end of the delivery? No. Strategy and Insights can help create surveys if needed Who is the owner of the data analysis for this experiment? kirill@getpocket.com Will this experiment require uplift? False QA Status of your code: QA Status: GREEN. For more information, see: https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-popular-topics-release-test-v3/#comment1113 Link to more information about this experiment: https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-popular-topics-release-test-v3/
activity-stream-pocket-new-tab-popular-topics-release-test-v3
New Tab Popular Topics Release Experiment
Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.
Data Science Issue: https://jira.mozilla.com/browse/DO-352
More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-popular-topics-release-test-v3/
Comment 1•4 years ago
|
||
[Tracking Requested - why for this release]:
need to relaunch this experiment now that telemetry pipeline is fixed: Bug 1657666
Updated•4 years ago
|
Comment 2•4 years ago
|
||
New Tab Layout Study - Popular Topics - Firefox Release 80 (9/1))
Firefox Release 80
We have finished testing the New Tab Layout Study - Popular Topics - Firefox Release 80 (9/1) experiment.
QA’s recommendation: GREEN-SHIP IT
Why is this experiment green?
- We haven’t found any new issues during testing and the known issues do not affect end users in a harmful way.
Testing Summary:
- Full Functional test suite: TestRail
Tested Platforms:
- Windows 10 x64
- macOS 10.15.6
- Linux MX 4.19 x64
Tested Firefox versions:
- Firefox Beta 80 en-US (BuildID 20200818235255)
- Firefox Nightly 81 en-US (BuildID 20200824094458)
Reporter | ||
Updated•4 years ago
|
Reporter | ||
Comment 3•4 years ago
|
||
Start Date: 2020-09-01 End Date: 2020-10-06
Reporter | ||
Updated•4 years ago
|
Updated•4 years ago
|
Description
•