[Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-personalization-v3 Fx 84.0 to 85.0 Release
Categories
(Shield :: Shield Study, task, P3)
Tracking
(firefox84+ fixed)
People
(Reporter: experimenter, Assigned: kdemtchouk)
References
()
Details
(Whiteboard: [no-nag])
User Story
Delivery Type: Pref Flip Experiment
What is the preference we will be changing
browser.newtabpage.activity-stream.discoverystream.personalization.version
What are the branches of the experiment and what values should
each branch be set to?
- Treatment variant 50%:
Value: None
Spocs are recommended using interest topics
- Control control 50%:
Value: None
Spocs are recommended using domain affinities
What version and channel do you intend to ship to?
1% of Release Firefox 84.0 to 85.0
Are there specific criteria for participants?
Also:
common Pocket Filter 1
Countries: United States of America (US)
Locales: English (British) (en-GB) English (Canadian) (en-CA) English (US) (en-US)
What is your intended go live date and how long will the experiment run?
Dec 15, 2020 - Feb 09, 2021 (56 days)
What is the main effect you are looking for and what data will you use to
make these decisions?
The main effect we're looking for is whether users will engage more with sponsored content items relevant to the general topics they are interested in.
Our main success metrics are:
1. Higher CTR on personalized SPOCs versus non-personalized SPOCs.
2. Meeting minimum impression targets on ad flights prioritized in the test variant.
3. Lower dismiss rate on personalized SPOCs versus non-personalized.
Do you plan on surveying users at the end of the delivery? No. Strategy and Insights can help create surveys if needed
Who is the owner of the data analysis for this experiment?
kirill@getpocket.com
Will this experiment require uplift?
False
QA Status of your code:
Requested: https://jira.mozilla.com/browse/PI-873
Link to more information about this experiment:
https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-personalization-v3/
activity-stream-pocket-new-tab-personalization-v3
Pocket New Tab SPOCs v3
Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.
Data Science Issue: https://jira.mozilla.com/browse/DO-352
More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-personalization-v3/
|
||
Comment 1•4 years ago
|
||
[Tracking Requested - why for this release]:
Launching this experiment on 12/15 with Fx 84, with plans to roll out to 50% in Jan, and if all goes well, turning on this feature to 100% w/ Fx 85.
|
||
Updated•4 years ago
|
|
||
Updated•4 years ago
|
|
||
Comment 2•4 years ago
|
||
activity-stream-pocket-new-tab-personalization-v3
Release Firefox 84.0 to 85.0
We have finished testing the [PI-873] Experiment: [Pocket + New Tab] SPOC Targeting V3 - Release 84 - 12/15 experiment.
Quality status: GREEN - SHIP IT
Why is this experiment green?
-
Delivery mechanism:
- No issues found.
-
Product specific:
- The only major issue found - Bug 1680791 - has been verified as fixed. It was not caused by this experiment, but by other experiments found on the stage environment.
Testing Summary:
- Test Runs: TestRail.
- The test recipe used “https://spocs.getpocket.com/spocs?country=US®ion=CA&site=1070098” instead of “https://spocs.getpocket.com/spocs&site=1070098” as value for “browser.newtabpage.activity-stream.discoverystream.spocs-endpoint” in order for us to test the experiment without using a VPN, after talking about this with Scott Downe.
Tested Platforms:
- Windows 10
- Linux Mint 20
- macOS 10.15
Tested Firefox versions:
- Firefox 84.0 RC en-CA, en-US, en-GB (Build ID: 20201207203640)
- Firefox 84.0 Beta 8 en-GB (Build ID: 20201203211213) (test run and update)
- Firefox 84.0 Beta 7 en-GB (Build ID: 20201201213706)
Regards,
Maria Heres
Ecosystem QA Team
|
Reporter | |
Updated•4 years ago
|
|
|
Reporter | |
Comment 3•4 years ago
|
||
|
|
Reporter | |
Comment 4•4 years ago
|
||
|
|
Reporter | |
Updated•4 years ago
|
|
|
||
Updated•4 years ago
|
Description
•