[Experiment] Add-On Experiment: Search Interventions Fx 72.0 to 74.0 Release
Categories
(Shield :: Shield Study, task, P3)
Tracking
(firefox72+ wontfix, firefox73+ wontfix, firefox74+ wontfix)
People
(Reporter: experimenter, Assigned: mconnor)
References
()
Details
User Story
Delivery Type: Add-on experiment
What are the branches of the experiment:
- Treatment treatment 50%:
When User types queries, we will show different search tips.
- Control control 50%:
Standard address bar experience
What version and channel do you intend to ship to?
2% of Release Firefox 72.0 to 74.0
Are there specific criteria for participants?
Countries: all
Locales: English (Australian) (en-AU) English (British) (en-GB) English (Canadian) (en-CA) English (New Zealand) (en-NZ) English (South African) (en-ZA) English (US) (en-US)
What is your intended go live date and how long will the experiment run?
Feb 12, 2020 - Apr 15, 2020 (63 days)
What is the main effect you are looking for and what data will you use to
make these decisions?
Primary effect is an increase in tagged search share. Tagged Search Share is the percentage of total search (tagged+organic) going through our access points:
Formula is:
tagged / (tagged + organic)
We also have SABER to conduct automated analysis of this experiment.
Who is the owner of the data analysis for this experiment?
mmccorquodale@mozilla.com
Will this experiment require uplift?
False
QA Status of your code:
QA finished the TestRail cases on 12/12/2019. Testing is ongoing.
Sign off as YELLOW as per comment: https://experimenter.services.mozilla.com/experiments/search-interventions/#comment590
bkal:
The Logo is fixed
We are ok with the survey display issue discussed, as it is designed we are ok with moving forward.
Link to more information about this experiment:
https://experimenter.services.mozilla.com/experiments/search-interventions/
Search Interventions
blah
Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.
More information: https://experimenter.services.mozilla.com/experiments/search-interventions/
|
||
Comment 1•4 years ago
|
||
Search Experiment: Interventions
Firefox Release 72.x
We have finished testing the Search Experiment: Interventions experiment.
Quality status: YELLOW - SHIP IT CONDITIONALLY
Why is this feature yellow?
- During testing, we uncovered two new issues. While both the issues have minor severity, Bug 1609139 should be a high priority one due to the fact that we are using the old Firefox logo in the presented survey.
- We haven’t received a confirmation to our uncertainty, that if it’s intended for all the tips to trigger the same survey. And if the current survey that we received for testing is the final version.
What needs to be done?
- Bug 1609139 should be fixed since the Firefox icon from the survey is outdated.
- Clarify if the same survey should be triggered by all of the tips and if it is the final version.
Testing Summary:
- Test Suite: Test Rail;
Tested Platforms:
- Windows 10 x64;
- Mac 10.15;
- Debian 9;
Tested Firefox versions:
- Firefox Dev Edition 72.0b10 en- US;
- Firefox Dev Edition 72.0b10 en-GB;
- Firefox Dev Edition 72.0b10 en-CA;
- Firefox 72.0 Unbranded build;
|
||
Comment 2•4 years ago
|
||
[Tracking Requested - why for this release]:
|
|
||
Comment 3•4 years ago
|
||
Search Experiment: Interventions for Firefox 73
Firefox Release 73.x
We have finished testing the Search Experiment: Interventions experiment for Firefox 73. The study is not live yet, but since it will run for 63 days, we are running our test suite on the latest build of each Beta cycle to make sure that there are no new issues between Firefox versions.
Quality status: YELLOW
Why is this feature yellow?
- During testing, we uncovered two new issues. While both the issues have minor severity, Bug 1609139 should be a high priority one due to the fact that we are using the old Firefox logo in the presented survey.
- We haven’t received a confirmation to our uncertainty, that if it’s intended for all the tips to trigger the same survey. And if the current survey that we received for testing is the final version.
What needs to be done?
- Bug 1609139 should be fixed since the Firefox icon from the survey is outdated.
- Clarify if the same survey should be triggered by all of the tips and if it is the final version.
Testing Summary:
Tested Platforms:
- Windows 10 x64;
- Mac 10.15;
- Debian 9;
Tested Firefox versions:
- Firefox Dev Edition 73.0b10 en-US;
- Firefox Dev Edition 73.0b10 en-GB;
- Firefox Dev Edition 73.0b10 en-CA;
|
Reporter | |
Updated•4 years ago
|
|
||
Updated•4 years ago
|
|
|
Reporter | |
Comment 4•4 years ago
|
||
Start Date: 2020-02-11 End Date: 2020-04-14
|
|
||
Comment 5•4 years ago
|
||
Search Experiment: Interventions
Firefox Release 74.x
We have finished testing the Search Experiment: Interventions experiment for Firefox 74. The study is already live, but since it's running for 63 days, we are running our test suite on the latest builds of each Beta cycle to make sure that there are no new issues between Firefox versions.
Quality status: GREEN
Why is this feature green?
- During testing, we haven’t found any issues.
Testing Summary:
Tested Platforms:
- Windows 10 x64;
- Mac 10.15;
- Debian 9;
Tested Firefox versions:
- Firefox Beta 74.0b8 en-US;
- Firefox Beta 74.0b8 en-GB;
- Firefox Beta 74.0b8 en-CA;
|
|
Reporter | |
Updated•4 years ago
|
Description
•