Closed Bug 1701002 Opened 5 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: Experiment system test of neutral change to control V3 Fx 87.0 to 87.0 Nightly

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: experimenter, Assigned: shell)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

None

    What are the branches of the experiment and what values should
    each branch be set to?

- Control Control with default + no-op change 50%:

Value: None

Control branch will have one pref that does nothing (boolean) - default.  And one pref (the string) that actively changes - which also does nothing.
        
- Treatment treatment1 50%:

Value: None

both prefs change from default - both do nothing
        

    What version and channel do you intend to ship to?

30% of Nightly Firefox 87.0 to 87.0

    Are there specific criteria for participants?

Experiments:
Avoid https://experimenter.services.mozilla.com/experiments/experiment-system-test-of-neutral-change-to-control/
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Mar 25, 2021 - Apr 15, 2021 (21 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

We will measure success by a non-statistical difference between branches using partybal.

We will measure success by a non-visible difference between branches in grafana dashboard.

    Who is the owner of the data analysis for this experiment?

None

    Will this experiment require uplift?

False

    QA Status of your code:

no QA needed.  same no-op prefs as before (and no QA was needed to use those).  We will ask for QA if this works and we decide to do this larger scale on release experiments.

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/experiment-system-test-of-neutral-change-to-control-v3/

Experiment system test of neutral change to control V3

Checking if adding a pref change that does nothing in the control branch - cause similar slight unenrollment that we see in treatment branches. This is to improve reliability of learnings by removing any bias from control branches.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-453
More information: https://experimenter.services.mozilla.com/experiments/experiment-system-test-of-neutral-change-to-control-v3/

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Experiment system test of neutral change to control V3 → [Experiment] Pref-Flip Experiment: Experiment system test of neutral change to control V3 Fx 87.0 to 87.0 Nightly
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.