Closed Bug 1700407 Opened 4 years ago Closed 4 years ago

[Experiment] Pref-Flip Experiment: Experiment system test of neutral change to control Fx 87.0 to 87.0 Nightly

Categories

(Shield :: Shield Study, task, P3)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: experimenter, Assigned: shell)

References

()

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

None

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment treatment2 33%:

Value: None

both prefs change from default - both do nothing
        
- Treatment treatment1 34%:

Value: None

both prefs change from default - both do nothing
        
- Control Control with default + no-op change 33%:

Value: None

Control branch will have one pref that does nothing (boolean) - default.  And one pref (the string) that actively changes - which also does nothing.
        

    What version and channel do you intend to ship to?

30% of Nightly Firefox 87.0 to 87.0

    Are there specific criteria for participants?

Prefs:

Experiments:

Any additional filters:
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Mar 23, 2021 - Apr 13, 2021 (21 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

We will measure success by a non-statistical difference between branches using partybal.

We will measure success by a non-visible difference between branches in grafana dashboard.

    Who is the owner of the data analysis for this experiment?

None

    Will this experiment require uplift?

False

    QA Status of your code:

no QA needed.  same no-op prefs as before (and no QA was needed to use those).  We will ask for QA if this works and we decide to do this larger scale on release experiments.

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/experiment-system-test-of-neutral-change-to-control/

Experiment system test of neutral change to control

Checking if adding a pref change that does nothing in the control branch - cause similar slight unenrollment that we see in treatment branches. This is to improve reliability of learnings by removing any bias from control branches.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-453
More information: https://experimenter.services.mozilla.com/experiments/experiment-system-test-of-neutral-change-to-control/

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Experiment system test of neutral change to control → [Experiment] Pref-Flip Experiment: Experiment system test of neutral change to control Fx 87.0 to 87.0 Nightly
Start Date: 2021-03-23 End Date: 2021-04-13
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.