Open Bug 1573840 Opened 11 months ago Updated 15 days ago

Staged Rollout of DoH to US Users

Categories

(Firefox :: System Add-ons: Off-train Deployment, task)

Desktop
Unspecified
task
Not set
normal

Tracking

()

ASSIGNED
Firefox 69

People

(Reporter: wthayer, Assigned: wthayer)

References

(Depends on 4 open bugs, Blocks 2 open bugs)

Details

(Whiteboard: [trr] [go-faster-system-addon])

Attachments

(27 files, 22 obsolete files)

16.96 KB, application/x-xpinstall
Details
121.33 KB, application/x-xpinstall
Details
121.22 KB, application/x-xpinstall
Details
121.28 KB, application/x-xpinstall
Details
121.65 KB, application/x-xpinstall
Details
121.77 KB, application/x-xpinstall
Details
121.54 KB, application/x-xpinstall
Details
121.56 KB, application/x-xpinstall
Details
127.68 KB, application/x-xpinstall
Details
2.57 KB, text/plain
bmiroglio
: data-review+
Details
128.67 KB, application/x-xpinstall
Details
129.07 KB, application/x-xpinstall
Details
24.76 KB, application/x-xpinstall
Details
24.80 KB, application/x-xpinstall
Details
24.95 KB, application/x-xpinstall
Details
24.94 KB, application/x-xpinstall
Details
25.68 KB, application/x-xpinstall
Details
24.94 KB, application/x-xpinstall
Details
20.05 KB, application/zip
Details
24.92 KB, application/x-xpinstall
Details
25.68 KB, application/x-xpinstall
Details
20.07 KB, application/x-xpinstall
Details
25.69 KB, application/x-xpinstall
Details
20.07 KB, application/x-xpinstall
Details
25.69 KB, application/x-xpinstall
Details
20.10 KB, application/x-xpinstall
Details
25.73 KB, application/x-xpinstall
Details

We plan to stage a rollout of DNS over HTTPS to US users. The plan is to:

  1. Deploy asystem add-on via Balrog that (a) runs some heuristics to determine if DoH can be enabled without causing problems, (b) if so, turns on DoH and notifies the user via a doorhanger
  2. Use Normandy to enable the add-on for a small (TBD) percentage of users.
  3. Measure engagement for 30 days
  4. If engagement looks good, ramp up to 100% of US users
  5. Flip add-on pref for all US users in next dot release. Continue to allow the-add on to run heuristics on each restart and network change until the heuristics are moved into the platform

Target start date is Sept 24 2019

Target audience is Firefox 69 desktop, US Geolocation (Normandy)

The DoH feature has been in Firefox for some time and a number of experiments have been run, so this will follow the Low Risk playbook.

The name of the pref to be flipped to enable the add-on is TBD

See Also: → 1566924
Component: Networking: DNS → System Add-ons: Off-train Deployment
Product: Core → Firefox
Hardware: Unspecified → Desktop
Target Milestone: --- → Firefox 69

Release Note Request
[Why is this notable]: DNS over HTTPS is a security feature that enhances our user's privacy by encrypting DNS queries. In late September, we plan to begin enabling this feature for our users in the USA
[Affects]: Release desktop users in the US
[Suggested wording]: DNS over HTTPS protects your privacy by encrypting the names of the website you visit when Firefox is converting them to IP addresses. Mozilla will soon begin to enable this feature by default for users in the USA. Each user will be notified and given a chance to opt-out before DNS over HTTPS is enabled.
[Links (documentation, blog post, etc)]: https://support.mozilla.org/en-US/kb/firefox-dns-over-https

relnote-firefox: --- → ?

Removed relnote flag - decided not to include this in 69 release notes.

relnote-firefox: ? → ---
Depends on: 1577027
Depends on: 1577347
Attached file doh_roll-out-0.0.1.zip (obsolete) —

Is it possible to get this signed for QA testing please?

Flags: needinfo?(mcooper)
Attachment #9090079 - Attachment is obsolete: true
Flags: needinfo?(mcooper)

I've signed this for testing, however please note that I cannot sign this for release via Balrog, and there is generally a separate QA process during the Balrog release cycle.

For further Balrog release questions, I'll direct you toward Rehan Dalal instead of me.

Whiteboard: [trr] → [trr] [go-faster-system-addon]
Attached file doh_roll-out-0.0.2.zip (obsolete) —

This is the revised release we are requesting :mythmon sign for testing. Thank you!

Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
Depends on: 1579985
Attached file doh_roll-out-0.0.3.zip (obsolete) —

Another release for testing! I will batch additional issue fixes in any upcoming test signing requests but this specific issue was a blocker for the PI team.

References:

Attachment #9090055 - Attachment is obsolete: true
Attachment #9091040 - Attachment is obsolete: true
Flags: needinfo?(mcooper)

Realized that :mythmon: is out on PTO.

:rdalal: - Could you please sign this release for testing? We will hand it off to the PI team.

Flags: needinfo?(mcooper) → needinfo?(rdalal)
No longer depends on: 1579985
Flags: needinfo?(rdalal)
Attached file doh_roll-out-0.0.4.zip (obsolete) —

Please sign the following for testing.

This is Sept 12th's nightly release: v0.0.4.

After it's signed, this should be ready for the PI team to test/confirm telemetry reporting. We'll have one more cut coming end of day tomorrow for Monday PI testing. Thanks!

Attachment #9091874 - Attachment is obsolete: true
Flags: needinfo?(rdalal)

We will be cutting a final v.0.0.5 today, so we can disregard signing the previous nightly version.

Flags: needinfo?(rdalal)
Attached file doh_roll-out-0.0.5rc1.zip (obsolete) —

Here is the first of two different release candidates!

Flags: needinfo?(rdalal)
Attached file doh_roll-out-0.0.5rc2.zip (obsolete) —

Here is the second of two different release candidates!

Attachment #9092521 - Attachment is obsolete: true
Flags: needinfo?(rdalal)
Attached file doh_roll-out-0.0.5rc1.zip (obsolete) —

Revised manifest file to have same version number as zip (1 of 2)

Attachment #9092771 - Attachment is obsolete: true
Flags: needinfo?(rdalal)
Attached file doh_roll-out-0.0.5rc2.zip (obsolete) —

Revised manifest file to have same version number as zip (2 of 2)

Attachment #9092772 - Attachment is obsolete: true
Attachment #9092773 - Attachment is obsolete: true
Flags: needinfo?(rdalal)
Attached file doh_roll-out-0.0.6.zip (obsolete) —

Here's the latest nightly release!

Attachment #9092777 - Attachment is obsolete: true
Attachment #9092778 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
Attached file doh_roll-out-0.0.6-internal.zip (obsolete) —

This is a duplicate of the previous release with one change – the gated pref has been removed doh-rollout.enabled to streamline the install process for internal testing. This version is tagged as 0.0.6-internal.

Attachment #9093436 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
Attachment #9093653 - Attachment is obsolete: true
Attached file doh_roll-out-0.0.7.zip (obsolete) —

Version 0.0.7 – Nightly release for the PI team to test next Monday.

Attachment #9093651 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
Attached file Data collection review
Attachment #9094686 - Flags: data-review?(bmiroglio)
Attached file doh_roll-out-0.0.8.zip (obsolete) —

v0.0.8 release for testing!

Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
Comment on attachment 9094686 [details]
Data collection review

# Data Review Form

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way? (see [here](https://github.com/mozilla/activity-stream/blob/master/docs/v2-system-addon/data_dictionary.md), [here](https://github.com/mozilla-mobile/focus/wiki/Install-and-event-tracking-with-the-Adjust-SDK), and [here](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/index.html) for examples).  Refer to the appendix for "documentation" if more detail about documentation standards is needed.

Yes, this is documented in the DOH rollout repo: https://github.com/mozilla/doh-rollout/blob/master/docs/telemetry.md


2) Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available.

Yes, this is can be turned off through the Firefox Preferences.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

Wayne Thayer and Nhi Nguyen will permanently monitor this data.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2: Interaction Data

5) Is the data collection request for default-on or default-off?

default-on

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No,

7) Is the data collection covered by the existing Firefox privacy notice? 

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) 

No.

9) Does the data collection use a third-party collection tool? 

No.

data-review: r+
Attachment #9094686 - Flags: data-review?(bmiroglio) → data-review+
Attached file doh_roll-out-0.0.9-rc1.zip (obsolete) —
Attachment #9094317 - Attachment is obsolete: true
Attachment #9094927 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
Depends on: 1584613

They are uplifting fixes now - so the ETA will be to deploy next week and begin the experiment the following Tuesday Oct 15th https://experimenter.services.mozilla.com/experiments/doh-us-staged-rollout-engagement-study/#population-comments

Attached file doh_roll-out-0.0.10.zip (obsolete) —

Nightly release to fix #72 and #94.

Attachment #9097053 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Attached file doh_roll-out-0.0.10.zip (obsolete) —

Revised v10 release!

Attachment #9098390 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Attached file doh_roll-out-0.0.11.zip (obsolete) —

Nightly release!

Attachment #9098401 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
Depends on: 1586331
See Also: → 1586331
Comment on attachment 9094686 [details]
Data collection review

Hi Ben -- the telemetry collection has changed a little bit; the differences since the last review are here: https://github.com/mozilla/doh-rollout/compare/2da62485a6f24fca83025b9fbe02ce1e85aba2b4..fd49b3c62f06452071469c30ce13a3ba099d6385#diff-b6840e6fe12d2b4f3ce88beefb5c876f

Can you sign off on the new collections?
Attachment #9094686 - Flags: data-review+ → data-review?(bmiroglio)
Attached file doh_roll-out-0.0.12.zip (obsolete) —

Nightly release!

Attachment #9098683 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
Comment on attachment 9094686 [details]
Data collection review

# Data Review Form

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way? (see [here](https://github.com/mozilla/activity-stream/blob/master/docs/v2-system-addon/data_dictionary.md), [here](https://github.com/mozilla-mobile/focus/wiki/Install-and-event-tracking-with-the-Adjust-SDK), and [here](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/index.html) for examples).  Refer to the appendix for "documentation" if more detail about documentation standards is needed.

Yes, this is documented in the DOH rollout repo: https://github.com/mozilla/doh-rollout/blob/master/docs/telemetry.md


2) Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available.

Yes, this is can be turned off through the Firefox Preferences.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

Wayne Thayer and Nhi Nguyen will permanently monitor this data.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2: Interaction Data

5) Is the data collection request for default-on or default-off?

default-on

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No,

7) Is the data collection covered by the existing Firefox privacy notice? 

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) 

No.

9) Does the data collection use a third-party collection tool? 

No.

data-review: r+
Attachment #9094686 - Flags: data-review?(bmiroglio) → data-review+

DNS over HTTPS US Rollout Add-on
Firefox Release 69.0.x

We have finished testing the DNS over HTTPS US Rollout Add-on version 0.0.12.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • All the blocker issues were fixed and verified. The remaining open issues have low impact and are not affecting the users in a major way.
  • Since 69.0.3 is not yet released, we have verified the Parental Control scenarios against latest Firefox Beta 70.0b13. After pushing the add-on to the test channel, we will also verify it on Firefox Release 69.0.3 build.

Testing Summary:

Tested Platforms:

  • Windows 10 x64

Tested Firefox versions:

  • Firefox Release 69.0.2
  • Firefox Beta 70.0b13

Requesting approval from RelMan for shipping the add-on.

Flags: needinfo?(ryanvm)

Approved on behalf of RelMan for 69.0.3 pending QA's testing with that build.

Flags: needinfo?(ryanvm)
Attached file doh_roll-out-1.0.0.zip (obsolete) —

This is the US Rollout version – no changes have been made between it and v0.0.12, except the version number.

:Mythmon – Can you sign for full release?

Attachment #9099391 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)

request to deploy to test channel

Flags: needinfo?(rdalal)

The add-on that I signed cannot be used for Balrog deployment. It needs to be signed specifically for Balrog, and I can't do that.

Attached file signed.9100564.xpi

Signed file attached. Please test.

This XPI has been deployed on the test channel (release-sysaddon) and is staged on the live release channel pending relman sign off in Balrog.

Relman sign off should only be requested/granted after QA has verified that everything is working as expected on the test channel.

Flags: needinfo?(rdalal)

DNS over HTTPS US Rollout Add-on - Verification of Test Channel
Firefox Release 69.0.3

We have finished testing the DNS over HTTPS US Rollout Add-on - Verification of Test Channel.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • We haven’t found any issues during testing the installation through the test channel.
  • However, we are not sure if the deployment of the add-on in Balrog was supposed to be limited only to US users or not (in order to ease testing), since we're able to install the add-on via test channel and trigger the DoH doorhanger without using VPN for US.

Testing Summary:

  • Verified that the add-on correctly installs after setting the channel to “release-sysaddon”.
  • Verified that the {"step": "started", "addon_id": "doh-rollout@mozilla.org"} and {"step": "completed", "addon_id": "doh-rollout@mozilla.org"} events are generated.
  • Verified that DoH doorhanger is displayed after creating the “doh-rollout.enabled” pref and setting it to true.
  • Verified that Parental Controls are detected on the latest Firefox Release 69.0.3 build.

Tested Platforms:

  • Windows 10 x64

Tested Firefox versions:

  • Firefox Release 69.0.3

(In reply to Carmen Fat [:carmenf] - Experiments QA from comment #55)

  • However, we are not sure if the deployment of the add-on in Balrog was supposed to be limited only to US users or not (in order to ease testing), since we're able to install the add-on via test channel and trigger the DoH doorhanger without using VPN for US.

This is by design - balrog will deploy the add-on to all users and Normandy will limit the rollout to users in the US on Firefox 69.0.3 or higher.

Ryan: Can you review the Balrog changes for this launch?

Flags: needinfo?(ryanvm)

Signed off for RelMan in Balrog. The system addon is now live on release.

Flags: needinfo?(ryanvm)
Attached file doh_roll-out-1.0.1.zip (obsolete) —

Fast-follow testing release v1.0.1 — Please see attached!

Attachment #9100540 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)

The above signed XPI is suitable for testing by internal audiences, and shouldn't be widely deployed. It is not suitable for Balrog deployment. We'll need another signature from :wezhou for that once this release has been verified.

Version 1.0.2-rc1 of DoH Rollout Addon

Requesting internal testing signature for the PI team. This version addresses the default/user branch issue and adds the ZScaler canary.

Attachment #9101680 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
Depends on: 1590831

We have a green light from PI. Can someone deploy this version to the test channel?

(Note - This version was signed via the automation :mythmon set up!)

https://github.com/mozilla/doh-rollout/releases/download/1.0.2/doh_roll-out-1.0.2-signed.xpi

Flags: needinfo?(rdalal)
Flags: needinfo?(mcooper)

The signature made here is, like all the other signatures I've made, not suitable for deployment via Balrog. We'll need it to be resigned by :wezhou. After that I should be able to update the testing channel with the new version.

Wei: Can you sign v1.0.2 for Balrog deployment? Also, if you'd like I can look into automatically signing these artifacts for Balrog deployment as well, with clearance from the Autograph team.

Flags: needinfo?(wezhou)
Attached file signed.9103102.xpi

Wei: Can you sign v1.0.2 for Balrog deployment? Also, if you'd like I can look into automatically signing these artifacts for Balrog deployment as well, with clearance from the Autograph team.

Signed file attached, please test it.

Michael, if you can sign these automatically, that will be doing a favor for me, so you're more than welcome to do that, and thanks! :)

Flags: needinfo?(wezhou)

Wei: sorry, but it looks like you signed 1.0.2-rc1, from comment 63. Can you sign 1.0.2, which is linked from comment 64? That's the one that's actually ready to release.

Flags: needinfo?(wezhou)
Attached file doh_roll-out-1.0.2.xpi

:wezhou, please sign this version.

Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
Attached file signed.9104365.xpi

Signed file attached. Please test it.

Flags: needinfo?(wezhou)

This XPI has been deployed on the test channel (release-sysaddon) and is staged on the live release channel pending relman sign off in Balrog.

Relman sign off should only be granted after QA has verified that everything is working as expected on the test channel.

Flags: needinfo?(rdalal)
Flags: needinfo?(mcooper)

I've updated the test channel to include the package on Firefox 70 and 71.

Attached file doh_roll-out-1.0.3.xpi

This is the revised XPI (unsigned) that resolves the reported PI team blocking issues from this morning. Can we get this signed for release, please?

Flags: needinfo?(wezhou)
Flags: needinfo?(mcooper)
Attached file signed.9104766.xpi

Signed file attached. Please test.

Flags: needinfo?(wezhou)

This XPI has been deployed on the test channel (release-sysaddon) and is staged on the live release channel pending relman sign off in Balrog. The rules also include Firefox 70 and 71.

Relman sign off should only be granted after QA has verified that everything is working as expected on the test channel.

Flags: needinfo?(mcooper)
Attached file doh_roll-out-1.0.4.xpi

DoH Rollout Addon Version 1.0.4

Please see attached XPI (unsigned) that resolves the reported PI team blocking issue. Can we get this signed for release, please?

Flags: needinfo?(wezhou)
Flags: needinfo?(mcooper)
Attached file signed.9105074.xpi

Signed file attached. Please test.

Flags: needinfo?(wezhou)

1.0.4 has been updated on the testing channel, and this time it has been verified that the rules for Firefox 70 are working.

Flags: needinfo?(mcooper)
Depends on: 1595951

Issues with the balrog deployment model were discovered on 11/1. Today we plan to turn down the balrog deployment and replace it with a Normandy deployment of the rollout add-on.

I have updated the Balrog rule to remove this add-on from all versions. Rehan and I tested this change manually on the testing channel.

Liz, can you review this change to remove the add-on from Balrog and approve?

Flags: needinfo?(lhenry)

Signed off on the 4 rules in balrog for the release channel.

Flags: needinfo?(lhenry)
Depends on: 1598218
Depends on: 1599650
Depends on: 1603779
Depends on: 1613454
Depends on: 1613481
See Also: → 1613481
Depends on: 1613489
See Also: → 1613489
Depends on: 1613790
[Experiment]DoH NIGHTLY Rollout to All US Desktop Users status has been changed to: Ship
            url:https://experimenter.services.mozilla.com/experiments/doh-nightly-rollout-to-all-us-desktop-users/
[Experiment]DoH NIGHTLY Rollout to All US Desktop Users status has been changed to: Live
            url:https://experimenter.services.mozilla.com/experiments/doh-nightly-rollout-to-all-us-desktop-users/

Some Nightly users in the rollout in comment #82 may experience losing network connectivity. This is a known problem - workaround documented in https://bugzilla.mozilla.org/show_bug.cgi?id=1610836#c14.

Blocks: doh-rollout
No longer depends on: doh-rollout
You need to log in before you can comment on or make changes to this bug.