Closed
Bug 1700407
Opened 4 years ago
Closed 4 years ago
[Experiment] Pref-Flip Experiment: Experiment system test of neutral change to control Fx 87.0 to 87.0 Nightly
Categories
(Shield :: Shield Study, task, P3)
Shield
Shield Study
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: experimenter, Assigned: shell)
References
()
Details
User Story
Delivery Type: Pref Flip Experiment What is the preference we will be changing None What are the branches of the experiment and what values should each branch be set to? - Treatment treatment2 33%: Value: None both prefs change from default - both do nothing - Treatment treatment1 34%: Value: None both prefs change from default - both do nothing - Control Control with default + no-op change 33%: Value: None Control branch will have one pref that does nothing (boolean) - default. And one pref (the string) that actively changes - which also does nothing. What version and channel do you intend to ship to? 30% of Nightly Firefox 87.0 to 87.0 Are there specific criteria for participants? Prefs: Experiments: Any additional filters: Countries: all Locales: all What is your intended go live date and how long will the experiment run? Mar 23, 2021 - Apr 13, 2021 (21 days) What is the main effect you are looking for and what data will you use to make these decisions? We will measure success by a non-statistical difference between branches using partybal. We will measure success by a non-visible difference between branches in grafana dashboard. Who is the owner of the data analysis for this experiment? None Will this experiment require uplift? False QA Status of your code: no QA needed. same no-op prefs as before (and no QA was needed to use those). We will ask for QA if this works and we decide to do this larger scale on release experiments. Link to more information about this experiment: https://experimenter.services.mozilla.com/experiments/experiment-system-test-of-neutral-change-to-control/
Experiment system test of neutral change to control
Checking if adding a pref change that does nothing in the control branch - cause similar slight unenrollment that we see in treatment branches. This is to improve reliability of learnings by removing any bias from control branches.
Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.
Data Science Issue: https://jira.mozilla.com/browse/DO-453
More information: https://experimenter.services.mozilla.com/experiments/experiment-system-test-of-neutral-change-to-control/
Reporter | ||
Updated•4 years ago
|
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Experiment system test of neutral change to control → [Experiment] Pref-Flip Experiment: Experiment system test of neutral change to control Fx 87.0 to 87.0 Nightly
Reporter | ||
Comment 1•4 years ago
|
||
Start Date: 2021-03-23 End Date: 2021-04-13
Reporter | ||
Updated•4 years ago
|
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•