1573840 - Staged Rollout of DoH to US Users

Status: NEW

(Firefox :: System Add-ons: Off-train Deployment, task)

Description

[Wayne Thayer]

We plan to stage a rollout of DNS over HTTPS to US users. The plan is to:

1. Deploy asystem add-on via Balrog that (a) runs some heuristics to determine if DoH can be enabled without causing problems, (b) if so, turns on DoH and notifies the user via a doorhanger
2. Use Normandy to enable the add-on for a small (TBD) percentage of users.
3. Measure engagement for 30 days
4. If engagement looks good, ramp up to 100% of US users
5. Flip add-on pref for all US users in next dot release. Continue to allow the-add on to run heuristics on each restart and network change until the heuristics are moved into the platform

Target start date is Sept 24 2019

Target audience is Firefox 69 desktop, US Geolocation (Normandy)

The DoH feature has been in Firefox for some time and a number of experiments have been run, so this will follow the Low Risk playbook.

The name of the pref to be flipped to enable the add-on is TBD

Comment 1

[Wayne Thayer]

Release Note Request
[Why is this notable]: DNS over HTTPS is a security feature that enhances our user's privacy by encrypting DNS queries. In late September, we plan to begin enabling this feature for our users in the USA
[Affects]: Release desktop users in the US
[Suggested wording]: DNS over HTTPS protects your privacy by encrypting the names of the website you visit when Firefox is converting them to IP addresses. Mozilla will soon begin to enable this feature by default for users in the USA. Each user will be notified and given a chance to opt-out before DNS over HTTPS is enabled.
[Links (documentation, blog post, etc)]: https://support.mozilla.org/en-US/kb/firefox-dns-over-https

Comment 2

[Wayne Thayer]

Removed relnote flag - decided not to include this in 69 release notes.

Comment 3

[Jonathan Kingston [:jkt] he/him]

Attachment: doh_roll-out-0.0.1.zip

Is it possible to get this signed for QA testing please?

Comment 4

[Jonathan Kingston [:jkt] he/him]

Latest code is all of https://github.com/mozilla/doh-rollout/ plus https://github.com/mozilla/doh-rollout/pull/35 which I set to :baku to review.

Comment 5

[Michael Cooper [:mythmon]]

Attachment: testing

Comment 6

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 7

[Michael Cooper [:mythmon]]

I've signed this for testing, however please note that I cannot sign this for release via Balrog, and there is generally a separate QA process during the Balrog release cycle.

For further Balrog release questions, I'll direct you toward Rehan Dalal instead of me.

Comment 8

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.2.zip

This is the revised release we are requesting :mythmon sign for testing. Thank you!

Comment 9

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 10

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.3.zip

Another release for testing! I will batch additional issue fixes in any upcoming test signing requests but this specific issue was a blocker for the PI team.

References:

• Github issue: https://github.com/mozilla/doh-rollout/issues/48
• New release: https://github.com/mozilla/doh-rollout/releases/tag/0.0.3

Comment 11

[Maxx Crawford [:maxx]]

Realized that :mythmon: is out on PTO.

:rdalal: - Could you please sign this release for testing? We will hand it off to the PI team.

Comment 12

[Rehan Dalal (Inactive)]

Attachment: [e-mail]

Comment 13

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.4.zip

Please sign the following for testing.

This is Sept 12th's nightly release: v0.0.4.

After it's signed, this should be ready for the PI team to test/confirm telemetry reporting. We'll have one more cut coming end of day tomorrow for Monday PI testing. Thanks!

• Release notes: https://github.com/mozilla/doh-rollout/releases/tag/0.0.4

Comment 14

[Maxx Crawford [:maxx]]

We will be cutting a final v.0.0.5 today, so we can disregard signing the previous nightly version.

Comment 15

[Rehan Dalal (Inactive)]

Attachment: [e-mail]

Comment 16

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.5rc1.zip

Here is the first of two different release candidates!

• Release notes: https://github.com/mozilla/doh-rollout/releases/tag/0.0.5rc1

Comment 17

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.5rc2.zip

Here is the second of two different release candidates!

• Release notes: https://github.com/mozilla/doh-rollout/releases/tag/0.0.5rc2

Comment 18

[Rehan Dalal (Inactive)]

Attachment: [e-mail]

Comment 19

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.5rc1.zip

Revised manifest file to have same version number as zip (1 of 2)

Comment 20

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.5rc2.zip

Revised manifest file to have same version number as zip (2 of 2)

Comment 21

[Rehan Dalal (Inactive)]

Attachment: [e-mail]

Comment 22

[Rehan Dalal (Inactive)]

Attachment: [e-mail]

Comment 23

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.6.zip

Here's the latest nightly release!

• Release notes: https://github.com/mozilla/doh-rollout/releases/tag/0.0.6

Comment 24

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 25

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.6-internal.zip

This is a duplicate of the previous release with one change – the gated pref has been removed doh-rollout.enabled to streamline the install process for internal testing. This version is tagged as 0.0.6-internal.

• Release note: https://github.com/mozilla/doh-rollout/releases/tag/0.0.6-internal

Comment 26

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 27

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 28

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.7.zip

Version 0.0.7 – Nightly release for the PI team to test next Monday.

• Full Release Notes: https://github.com/mozilla/doh-rollout/releases/tag/0.0.7

Comment 29

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 30

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Attachment: Data collection review

Comment 31

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.8.zip

v0.0.8 release for testing!

• Release notes: https://github.com/mozilla/doh-rollout/releases/tag/0.0.8

Comment 32

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 33

[Ben Miroglio [:bmiroglio]]

Comment on attachment 9094686 [details]
Data collection review

# Data Review Form

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way? (see [here](https://github.com/mozilla/activity-stream/blob/master/docs/v2-system-addon/data_dictionary.md), [here](https://github.com/mozilla-mobile/focus/wiki/Install-and-event-tracking-with-the-Adjust-SDK), and [here](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/index.html) for examples).  Refer to the appendix for "documentation" if more detail about documentation standards is needed.

Yes, this is documented in the DOH rollout repo: https://github.com/mozilla/doh-rollout/blob/master/docs/telemetry.md


2) Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available.

Yes, this is can be turned off through the Firefox Preferences.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

Wayne Thayer and Nhi Nguyen will permanently monitor this data.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2: Interaction Data

5) Is the data collection request for default-on or default-off?

default-on

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No,

7) Is the data collection covered by the existing Firefox privacy notice? 

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) 

No.

9) Does the data collection use a third-party collection tool? 

No.

data-review: r+

Comment 34

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.9-rc1.zip

Release notes - https://github.com/mozilla/doh-rollout/releases/tag/0.0.9rc1

Comment 35

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 36

[:shell escalante]

They are uplifting fixes now - so the ETA will be to deploy next week and begin the experiment the following Tuesday Oct 15th https://experimenter.services.mozilla.com/experiments/doh-us-staged-rollout-engagement-study/#population-comments

Comment 37

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.10.zip

Nightly release to fix #72 and #94.

• Release notes: https://github.com/mozilla/doh-rollout/releases/tag/0.0.10

Comment 38

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.10.zip

Revised v10 release!

Comment 39

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 40

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.11.zip

Nightly release!

• Github release notes: https://github.com/mozilla/doh-rollout/releases/tag/0.0.11

Comment 41

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 42

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Comment on attachment 9094686 [details]
Data collection review

Hi Ben -- the telemetry collection has changed a little bit; the differences since the last review are here: https://github.com/mozilla/doh-rollout/compare/2da62485a6f24fca83025b9fbe02ce1e85aba2b4..fd49b3c62f06452071469c30ce13a3ba099d6385#diff-b6840e6fe12d2b4f3ce88beefb5c876f

Can you sign off on the new collections?

Comment 43

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-0.0.12.zip

Nightly release!

• Github release notes - https://github.com/mozilla/doh-rollout/releases/tag/0.0.12

Comment 44

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 45

[Ben Miroglio [:bmiroglio]]

Comment on attachment 9094686 [details]
Data collection review

# Data Review Form

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way? (see [here](https://github.com/mozilla/activity-stream/blob/master/docs/v2-system-addon/data_dictionary.md), [here](https://github.com/mozilla-mobile/focus/wiki/Install-and-event-tracking-with-the-Adjust-SDK), and [here](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/index.html) for examples).  Refer to the appendix for "documentation" if more detail about documentation standards is needed.

Yes, this is documented in the DOH rollout repo: https://github.com/mozilla/doh-rollout/blob/master/docs/telemetry.md


2) Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available.

Yes, this is can be turned off through the Firefox Preferences.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

Wayne Thayer and Nhi Nguyen will permanently monitor this data.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2: Interaction Data

5) Is the data collection request for default-on or default-off?

default-on

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No,

7) Is the data collection covered by the existing Firefox privacy notice? 

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) 

No.

9) Does the data collection use a third-party collection tool? 

No.

data-review: r+

Comment 46

[Carmen Fat [:cfat] - Ecosystem QA]

DNS over HTTPS US Rollout Add-on
Firefox Release 69.0.x

We have finished testing the DNS over HTTPS US Rollout Add-on version 0.0.12.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

• All the blocker issues were fixed and verified. The remaining open issues have low impact and are not affecting the users in a major way.
• Since 69.0.3 is not yet released, we have verified the Parental Control scenarios against latest Firefox Beta 70.0b13. After pushing the add-on to the test channel, we will also verify it on Firefox Release 69.0.3 build.

Testing Summary:

• TestRail: Test Plan (https://testrail.stage.mozaws.net/index.php?/plans/view/20499)

Tested Platforms:

• Windows 10 x64

Tested Firefox versions:

• Firefox Release 69.0.2
• Firefox Beta 70.0b13

Comment 47

[Nhi Nguyen (:nhi)]

Requesting approval from RelMan for shipping the add-on.

Comment 48

[Ryan VanderMeulen [:RyanVM]]

Approved on behalf of RelMan for 69.0.3 pending QA's testing with that build.

Comment 49

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-1.0.0.zip

This is the US Rollout version – no changes have been made between it and v0.0.12, except the version number.

• Release note: https://github.com/mozilla/doh-rollout/releases/tag/1.0.0

:Mythmon – Can you sign for full release?

Comment 50

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 51

[Nhi Nguyen (:nhi)]

request to deploy to test channel

Comment 52

[Michael Cooper [:mythmon]]

The add-on that I signed cannot be used for Balrog deployment. It needs to be signed specifically for Balrog, and I can't do that.

Comment 53

[:wezhou]

Attachment: signed.9100564.xpi

Signed file attached. Please test.

Comment 54

[Rehan Dalal (Inactive)]

This XPI has been deployed on the test channel (release-sysaddon) and is staged on the live release channel pending relman sign off in Balrog.

Relman sign off should only be requested/granted after QA has verified that everything is working as expected on the test channel.

Comment 55

[Carmen Fat [:cfat] - Ecosystem QA]

DNS over HTTPS US Rollout Add-on - Verification of Test Channel
Firefox Release 69.0.3

We have finished testing the DNS over HTTPS US Rollout Add-on - Verification of Test Channel.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

• We haven’t found any issues during testing the installation through the test channel.
• However, we are not sure if the deployment of the add-on in Balrog was supposed to be limited only to US users or not (in order to ease testing), since we're able to install the add-on via test channel and trigger the DoH doorhanger without using VPN for US.

Testing Summary:

• Verified that the add-on correctly installs after setting the channel to “release-sysaddon”.
• Verified that the {"step": "started", "addon_id": "[e-mail]"} and {"step": "completed", "addon_id": "[e-mail]"} events are generated.
• Verified that DoH doorhanger is displayed after creating the “doh-rollout.enabled” pref and setting it to true.
• Verified that Parental Controls are detected on the latest Firefox Release 69.0.3 build.

Tested Platforms:

• Windows 10 x64

Tested Firefox versions:

• Firefox Release 69.0.3

Comment 56

[Wayne Thayer]

(In reply to Carmen Fat [:carmenf] - Experiments QA from comment #55)

• However, we are not sure if the deployment of the add-on in Balrog was supposed to be limited only to US users or not (in order to ease testing), since we're able to install the add-on via test channel and trigger the DoH doorhanger without using VPN for US.

This is by design - balrog will deploy the add-on to all users and Normandy will limit the rollout to users in the US on Firefox 69.0.3 or higher.

Comment 57

[Michael Cooper [:mythmon]]

Ryan: Can you review the Balrog changes for this launch?

Comment 58

[Ryan VanderMeulen [:RyanVM]]

Signed off for RelMan in Balrog. The system addon is now live on release.

Comment 59

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-1.0.1.zip

Fast-follow testing release v1.0.1 — Please see attached!

• Github release notes: https://github.com/mozilla/doh-rollout/releases/tag/1.0.1

Comment 60

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 61

[Michael Cooper [:mythmon]]

The above signed XPI is suitable for testing by internal audiences, and shouldn't be widely deployed. It is not suitable for Balrog deployment. We'll need another signature from :wezhou for that once this release has been verified.

Comment 62

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-1.0.2-rc1.zip

Version 1.0.2-rc1 of DoH Rollout Addon

Requesting internal testing signature for the PI team. This version addresses the default/user branch issue and adds the ZScaler canary.

• Github release notes: https://github.com/mozilla/doh-rollout/releases/tag/1.0.2rc1

Comment 63

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 64

[Maxx Crawford [:maxx]]

We have a green light from PI. Can someone deploy this version to the test channel?

(Note - This version was signed via the automation :mythmon set up!)

https://github.com/mozilla/doh-rollout/releases/download/1.0.2/doh_roll-out-1.0.2-signed.xpi

Comment 65

[Michael Cooper [:mythmon]]

The signature made here is, like all the other signatures I've made, not suitable for deployment via Balrog. We'll need it to be resigned by :wezhou. After that I should be able to update the testing channel with the new version.

Wei: Can you sign v1.0.2 for Balrog deployment? Also, if you'd like I can look into automatically signing these artifacts for Balrog deployment as well, with clearance from the Autograph team.

Comment 66

[:wezhou]

Attachment: signed.9103102.xpi

Wei: Can you sign v1.0.2 for Balrog deployment? Also, if you'd like I can look into automatically signing these artifacts for Balrog deployment as well, with clearance from the Autograph team.

Signed file attached, please test it.

Michael, if you can sign these automatically, that will be doing a favor for me, so you're more than welcome to do that, and thanks! :)

Comment 67

[Michael Cooper [:mythmon]]

Wei: sorry, but it looks like you signed 1.0.2-rc1, from comment 63. Can you sign 1.0.2, which is linked from comment 64? That's the one that's actually ready to release.

Comment 68

[Michael Cooper [:mythmon]]

Attachment: doh_roll-out-1.0.2.xpi

:wezhou, please sign this version.

Comment 69

[:wezhou]

Attachment: signed.9104365.xpi

Signed file attached. Please test it.

Comment 70

[Michael Cooper [:mythmon]]

This XPI has been deployed on the test channel (release-sysaddon) and is staged on the live release channel pending relman sign off in Balrog.

Relman sign off should only be granted after QA has verified that everything is working as expected on the test channel.

Comment 71

[Michael Cooper [:mythmon]]

I've updated the test channel to include the package on Firefox 70 and 71.

Comment 72

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-1.0.3.xpi

This is the revised XPI (unsigned) that resolves the reported PI team blocking issues from this morning. Can we get this signed for release, please?

• Github Release Notes: https://github.com/mozilla/doh-rollout/releases/tag/1.0.3

Comment 73

[:wezhou]

Attachment: signed.9104766.xpi

Signed file attached. Please test.

Comment 74

[Michael Cooper [:mythmon]]

This XPI has been deployed on the test channel (release-sysaddon) and is staged on the live release channel pending relman sign off in Balrog. The rules also include Firefox 70 and 71.

Relman sign off should only be granted after QA has verified that everything is working as expected on the test channel.

Comment 75

[Maxx Crawford [:maxx]]

Attachment: doh_roll-out-1.0.4.xpi

DoH Rollout Addon Version 1.0.4

Please see attached XPI (unsigned) that resolves the reported PI team blocking issue. Can we get this signed for release, please?

• Github Release Notes: https://github.com/mozilla/doh-rollout/releases/tag/1.0.4

Comment 76

[:wezhou]

Attachment: signed.9105074.xpi

Signed file attached. Please test.

Comment 77

[Michael Cooper [:mythmon]]

1.0.4 has been updated on the testing channel, and this time it has been verified that the rules for Firefox 70 are working.

Comment 78

[Wayne Thayer]

Issues with the balrog deployment model were discovered on 11/1. Today we plan to turn down the balrog deployment and replace it with a Normandy deployment of the rollout add-on.

Comment 79

[Michael Cooper [:mythmon]]

I have updated the Balrog rule to remove this add-on from all versions. Rehan and I tested this change manually on the testing channel.

Liz, can you review this change to remove the add-on from Balrog and approve?

Comment 80

[Liz Henry (:lizzard) (relman/hg->git project)]

Signed off on the 4 rules in balrog for the release channel.

Comment 81

[Experimenter Bot]

[Experiment]DoH NIGHTLY Rollout to All US Desktop Users status has been changed to: Ship
            url:https://experimenter.services.mozilla.com/experiments/doh-nightly-rollout-to-all-us-desktop-users/

Comment 82

[Experimenter Bot]

[Experiment]DoH NIGHTLY Rollout to All US Desktop Users status has been changed to: Live
            url:https://experimenter.services.mozilla.com/experiments/doh-nightly-rollout-to-all-us-desktop-users/

Comment 83

[Nhi Nguyen (:nhi)]

Some Nightly users in the rollout in comment #82 may experience losing network connectivity. This is a known problem - workaround documented in https://bugzilla.mozilla.org/show_bug.cgi?id=1610836#c14.

Comment 84

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee is inactive on Bugzilla, so the assignee is being reset.