[Experiment] Pref-Flip Experiment: TopSites Launch Phase 3 - GB Fx 83.0 to 85.0 Release
Categories
(Shield :: Shield Study, task, P3)
Tracking
(firefox83+ fixed, firefox84+ fixed)
People
(Reporter: experimenter, Assigned: bkaluzny)
References
(Blocks 1 open bug, )
Details
(Whiteboard: [no-nag])
User Story
Delivery Type: Pref Flip Experiment What is the preference we will be changing browser.topsites.useRemoteSetting What are the branches of the experiment and what values should each branch be set to? - Control Control 50%: Value: None Don't use Remote Settings to power the Top Sites - Treatment Treatment-admarketplace 50%: Value: None Changes Amazon tile to a URL and sends a background request to ad market place What version and channel do you intend to ship to? 40% of Release Firefox 83.0 to 85.0 Are there specific criteria for participants? Prefs: Experiments: Any additional filters: Countries: United Kingdom (GB) Locales: all What is your intended go live date and how long will the experiment run? Nov 17, 2020 - Feb 25, 2021 (100 days) What is the main effect you are looking for and what data will you use to make these decisions? Our partner (AdMarketplace) is monitoring the traffic we send their way with the ‘amazon.attributionURL’. They’ve contacted us right away before when they saw something strange. Their internal contacts are Mike DeBoer, Dao, and mconnor. Who is the owner of the data analysis for this experiment? tbrooks@mozilla.com Will this experiment require uplift? False QA Status of your code: https://jira.mozilla.com/browse/PI-859 Link to more information about this experiment: https://experimenter.services.mozilla.com/experiments/topsites-launch-phase-3-gb-v2/
TopSites Launch Phase 3 - GB
TopSites Redirect Launch
Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.
Data Science Issue: https://jira.mozilla.com/browse/DS-673
More information: https://experimenter.services.mozilla.com/experiments/topsites-launch-phase-3-gb-v2/
Assignee | ||
Comment 1•4 years ago
|
||
[Tracking Requested - why for this release]:
Updated•4 years ago
|
Comment 2•4 years ago
|
||
[Experiment] Sign Off for PI-859 - TopSites Launch Phase 3 - GB - (YELLOW)
TopSites Launch Phase 3 - GB
Release Firefox 83.0 to 85.0
We have finished testing the [PI-859] TopSites Launch Phase 3 - GB experiment.
Quality status: YELLOW - SHIP IT CONDITIONALLY
Why is this experiment yellow?
Delivery mechanism:
- No issues found.
Product specific:
- We have encountered 4 known issues, all of which have been resolved as won’t fix for Firefox 83 prior to our testing. They affect user experience, but they are not considered blocking.
- We have found 2 new issues, one of which has been fixed and verified and the other has been closed as won’t fix.
- The page accessed from the “Our sponsors & your privacy” option of the Sponsored tiles is still in its placeholder form. QA has been informed it will be ready by the time the experiment will be launched, but it has not been ready during our testing or at the time of this sign-off.
What needs to be done?
- [Engineering] [Product]: Verify that the issues marked as won’t fix for 83 have taken into account the existence of the experiment.
- [Product]: Make sure that the page accessed from “Our sponsors & your privacy” is populated with the correct content before the experiment is launched.
Testing Summary:
- Test runs: TestRail
- Exploratory testing performed on Linux Mint 20 related to enrollment, Sponsored Top Sites being displayed, them being dismissable and that visiting Amazon and eBay can still make the non-Sponsored version be displayed in Top Sites.
- Testing has been performed on the Stage environment. Due to the nature of the experiment, care should be taken so that the Production version has been set up with the same content as the Stage version of the top-sites Remote Settings collection.
Tested Platforms:
- Windows 10
- Linux Mint 20 (exploratory and bug verification)
- macOS 10.15 (exploratory and bug verification)
Tested Firefox versions:
- Firefox 83.0 RC 2 (Build ID: 20201112153044)
- Firefox 83.0 RC 1 (Build ID: 20201110140026)
- Firefox Nightly 84.0a1 (Build ID: 20201113015110) (update scenario)
Reporter | ||
Updated•4 years ago
|
Reporter | ||
Comment 3•4 years ago
|
||
Updated•4 years ago
|
Comment 4•4 years ago
|
||
TopSites Launch Phase 3 - GB
Release Firefox 83.0 to 86.0
We have finished testing the [PI-904] [Long running study] TopSites Launch Phase 3 - GB for Firefox 85 experiment for version 85.
The study is already live, but since it's running until Firefox 86, we are running our test suite on the latest builds of each Beta cycle to make sure that there are no new issues between Firefox versions. We took into account the changes introduced by PI-916 and tested using Prod Preview.
Quality status: GREEN
Why is this experiment green?
- Delivery mechanism:
- No issues found.
- Product specific:
- We encountered the following known issues (https://mzl.la/39PJCMu) which do not affect user experience in a harmful way.
Testing Summary:
- Test runs: https://testrail.stage.mozaws.net/index.php?/plans/view/38981
- Some scenarios which would have normally be tested in this PI were tested in PI-916 (https://jira.mozilla.com/browse/PI-916).
Tested Platforms:
- Windows 10 x64
Tested Firefox versions:
- Firefox en-GB RC 85.0 (Build ID: 20210118153634)
Comment 5•4 years ago
|
||
We have finished testing the experiment on Firefox 86 and sent a GREEN sign off. You can find it here.
Comment 7•4 years ago
•
|
||
Assignee | ||
Comment 8•4 years ago
|
||
dao please close. I believe this work is complete, the experiment was launched.
Updated•4 years ago
|
Comment 9•3 years ago
|
||
We have finished testing the experiments for Firefox 89 and sent the sign of email. You can find it here: https://docs.google.com/document/d/1hse0eWQh89nYNUJ37VvSKbFabCY1LrOwXLC83-tnQwg/edit.
Updated•3 years ago
|
Comment 10•3 years ago
|
||
Description
•