Closed Bug 1577347 Opened 6 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: DoH US Engagement Study Fx 69.0 to 70.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox69+ fixed, firefox70+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox69 + fixed
firefox70 + fixed

People

(Reporter: experimenter, Assigned: wthayer)

References

(Blocks 1 open bug,
URL
)

Details

User Story

Experiment Type: Pref Flip Experiment

    What is the preference we will be changing

doh-rollout.enabled

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment enabled 50%:

Value: true

enabled
        
- Treatment disabled 50%:

Value: false

disabled
        

    What version and channel do you intend to ship to?

1% of Release Firefox 69.0 to 70.0

    Are there specific criteria for participants?

- 69.0.3 or higher (including 70.*)
- Enrollment should be sticky over country
- System addon doh-rollout@mozilla.org is installed
Countries: United States of America (US) 

Locales: Chinese (Simplified) (zh-CN) English (British) (en-GB) English (Canadian) (en-CA) English (US) (en-US) French (fr) German (de) German (Germany) (de-DE) Italian (it) Russian (ru) Spanish (Mexico) (es-MX) Spanish (Spain) (es-ES) 

    What is your intended go live date and how long will the experiment run?

Oct 17, 2019 - Nov 21, 2019 (35 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

We will be analyzing telemetry for anomalies as we slowly enable DoH, and also watching user sentiment on social media and in bug reports. This experiment will supplement these other feedback channels with engagement data. If engagement hold steady, we will continue to roll out DoH. If it drops, we will stop and possibly roll back the change.

Do you plan on surveying users at the end of the experiment? No.
Strategy and Insights can help create surveys if needed

    Who is the owner of the data analysis for this experiment?

tdsmith@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

The addon that this pref will enable is being tested at https://jira.mozilla.com/browse/PI-244

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/doh-us-staged-rollout-engagement-study/

DoH US Staged Rollout Engagement Study

We plan to conduct a study to monitor engagement and retention in conjunction with a staged rollout of DNS over HTTPS to US users.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/doh-us-staged-rollout-engagement-study/

[Tracking Requested - why for this release]: We're targeting an October 8 launch.

tracking-firefox69: --- → ?

Is this still the case? I don't see it in this week's launches in Experimenter.

Flags: needinfo?(wthayer)

This has been delayed to Oct 15.

Flags: needinfo?(wthayer)

DNS over HTTPS US Rollout Add-on - Verification of Test Channel
Firefox Release 69.0.3

We have finished testing the DNS over HTTPS US Rollout Add-on - Verification of Test Channel.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • We haven’t found any issues during testing the installation through the test channel.
  • However, we are not sure if the deployment of the add-on in Balrog was supposed to be limited only to US users or not (in order to ease testing), since we're able to install the add-on via test channel and trigger the DoH doorhanger without using VPN for US.

Testing Summary:

  • Verified that the add-on correctly installs after setting the channel to “release-sysaddon”.
  • Verified that the {"step": "started", "addon_id": "doh-rollout@mozilla.org"} and {"step": "completed", "addon_id": "doh-rollout@mozilla.org"} events are generated.
  • Verified that DoH doorhanger is displayed after creating the “doh-rollout.enabled” pref and setting it to true.
  • Verified that Parental Controls are detected on the latest Firefox Release 69.0.3 build.

Tested Platforms:

  • Windows 10 x64

Tested Firefox versions:

  • Firefox Release 69.0.3
User Story: (updated)
Summary: [Experiment]: Pref-Flip: DoH US Staged Rollout Engagement Study → [Experiment] Pref-Flip Experiment: DoH US Engagement Study Fx 69.0 to 70.0 Release
Start Date: 2019-10-17 End Date: 2019-11-21
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.