Closed Bug 1703296 Opened 5 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: Experiment system test of neutral change to control V4 Fx 89.0 to 91.0 Nightly

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: experimenter, Assigned: shell)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

None

    What are the branches of the experiment and what values should
    each branch be set to?

- Control Control with default + no-op change 50%:

Value: None

Control branch will have one pref that does nothing (boolean) - default.  And one pref (the string) that actively changes - which also does nothing.
        
- Treatment treatment2 50%:

Value: None

both prefs change from default - both do nothing
        

    What version and channel do you intend to ship to?

30% of Nightly Firefox 89.0 to 91.0

    Are there specific criteria for participants?

Prefs:

Experiments:

Any additional filters:
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Apr 06, 2021 - Apr 27, 2021 (21 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

We will measure success by a non-statistical difference between branches using partybal.

We will measure success by a non-visible difference between branches in grafana dashboard.

    Who is the owner of the data analysis for this experiment?

None

    Will this experiment require uplift?

False

    QA Status of your code:

no QA needed.  same no-op prefs as before (and no QA was needed to use those).  We will ask for QA if this works and we decide to do this larger scale on release experiments.

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/experiment-system-test-of-neutral-change-to-control-v4/

Experiment system test of neutral change to control V4

Checking if adding a pref change that does nothing in the control branch - cause similar slight unenrollment that we see in treatment branches. This is to improve reliability of learnings by removing any bias from control branches.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-453
More information: https://experimenter.services.mozilla.com/experiments/experiment-system-test-of-neutral-change-to-control-v4/

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Experiment system test of neutral change to control V4 → [Experiment] Pref-Flip Experiment: Experiment system test of neutral change to control V4 Fx 89.0 to 91.0 Nightly
Start Date: 2021-04-06 End Date: 2021-04-27
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.