Closed Bug 1690684 Opened 4 years ago Closed 4 years ago

[Experiment] Pref-Flip Experiment: unenrollment normandy test with no-op pref in Firefox - boolean V2 Fx 87.0 to 87.0 Nightly

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox87+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox87 + fixed

People

(Reporter: experimenter, Assigned: mythmon)

References

(
URL
)

Details

(Whiteboard: [no-nag])

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

app.normandy.test-prefs.bool

    What are the branches of the experiment and what values should
    each branch be set to?

- Control control boolean 50%:

Value: false

control leaving boolean at default false
        
- Treatment treatment boolean 50%:

Value: true

treatment changing default value from false to true
        

    What version and channel do you intend to ship to?

30% of Nightly Firefox 87.0 to 87.0

    Are there specific criteria for participants?

Firefox Build ID that has bug 1690052's patch in it: `>= "20210203214546 "`
Use a bucket sample range that excludes the previous boolean no-op test.
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Feb 09, 2021 - Feb 23, 2021 (14 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

look for uneven un-enrollment in a case there should be none.

We'll check the new meta data on standard normandy telemetry that says which pref caused the change landed in 85.

    Who is the owner of the data analysis for this experiment?

tismith@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

Discussed in Lightning advising with QA and DS (Krupa and Tim).  This is just no-op prefs on Nightly + Beta for system test.  Engineer QA is sufficient for this minimal risk.

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/unenrollment-normandy-test-with-no-op-pref-in-firefox-boolean-v2/

unenrollment normandy test with no-op pref in Firefox - boolean V2

Changing a pref that does nothing - to check enrollment and unenrollment reliability.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-397
More information: https://experimenter.services.mozilla.com/experiments/unenrollment-normandy-test-with-no-op-pref-in-firefox-boolean-v2/

[Tracking Requested - why for this release]:
Running a second no-op experiment to test bug 1690052's fix.

tracking-firefox87: --- → ?
Depends on: 1690052
status-firefox87: --- → affected
tracking-firefox87: ? → +
Whiteboard: [no-nag]
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: unenrollment normandy test with no-op pref in Firefox - boolean V2 → [Experiment] Pref-Flip Experiment: unenrollment normandy test with no-op pref in Firefox - boolean V2 Fx 87.0 to 87.0 Nightly
Start Date: 2021-02-09 End Date: 2021-02-23
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
status-firefox87: affected → fixed
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.