1631609 - DoH Rollout Extension: Steering heuristics

Status: RESOLVED FIXED

(Firefox :: Security, task, P1)

Description

[Valentin Gosu [:valentin]]

+++ This bug was initially created as a clone of Bug #1626057 +++

We need the rollout addon to be able to query a DNS name and map the CNAME response to one of the TRR URIs in our list.
That URI should be used for that network, if no user choices are present that say it shouldn't be used.

Comment 1

[Valentin Gosu [:valentin]]

Attachment: Bug 1631609 - [WIP] doh-rollout steering detection r=dragana

Comment 2

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: Bug 1631609 - Steer to network-indicated DoH endpoint if detected. r=valentin!,johannh!

Comment 3

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: Bug 1631609 - Stop mocking DoH heuristics for tests. r=johannh!

Comment 4

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: Bug 1631609 - Use an actual URI for dummy TRR in tests. r=johannh!,valentin!

This ensures we can check gDNSService.currentTrrURI

Comment 5

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: Bug 1631609 - Test provider steering heuristic. r=valentin!,johannh!

Comment 6

[(inactive) Nihanth Subramanya [:nhnt11]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=96e277ba6b15be67c2bf082fc991d698ba20e093

Comment 7

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: Bug 1631609 - Steer to network-indicated DoH endpoint if detected (patch for beta). r=valentin!,johannh!

This patch was created by rebasing the patches for central onto beta and resolving conflicts. Also some small modifications to accommodate for the lack of auto-trr selection committing.

Comment 8

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: Folded patch for beta

This patch is basically a folded version of the patches for central with some modifications to account for the lack of auto-trr-selection committing in 77. Attaching as a patch on the bug in addition to phabricator because we need different patches for beta and central and I don't know how to do that with Lando.

Comment 9

[(inactive) Nihanth Subramanya [:nhnt11]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=91629fd0fc61cac80fd42c2dfb4505c8bc369048
https://treeherder.mozilla.org/#/jobs?repo=try&revision=3954cc586cbba88a7b3965c5d5612859546260f4

Comment 10

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: Folded patch for beta v2

Same as comment 8 but added in the fix for removing the doorhanger tab progress listener.

Comment 11

[Natalia Csoregi [:nataliaCs]]

Backed out for multiple doh-rollout failures

backout: https://hg.mozilla.org/integration/autoland/rev/835bd05cf1352b33b3d3f5ce31dd0d204a4ecc51

push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=87f51b855906deebc45dbe3278b502433a9d60c6&group_state=expanded&selectedTaskRun=FkCDd3yFS7af9mXgChr9IQ-0

failure logs:

• browser/extensions/doh-rollout/test/browser/browser_policyOverride.js https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=305124717&repo=autoland&lineNumber=4281
• browser/extensions/doh-rollout/test/browser/browser_trrSelection_disable.js https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=305119818&repo=autoland&lineNumber=5541
• browser/extensions/doh-rollout/test/browser/browser_providerSteering.js https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=305124236&repo=autoland&lineNumber=3062

Comment 12

[Natalia Csoregi [:nataliaCs]]

Backed out for failures on browser_providerSteering.js.

backout: https://hg.mozilla.org/integration/autoland/rev/d4f826731c36152a3e9612413cc8e74f9cc73a86

push: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C10%2Cx64%2Casan%2Cmochitests%2Ctest-windows10-64-asan%2Fopt-mochitest-browser-chrome-e10s-3%2Cm%28bc3%29&tochange=889d7cd14e4d76462eabe4a26a551c72fb416e86&fromchange=fecffba489bd447d60c2930f9d55ef834b1b20da&selectedTaskRun=SnZfnbi8Qw-KIStIs1KAXA-0

failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=305157943&repo=autoland&lineNumber=3302

[task 2020-06-05T02:59:02.905Z] 02:59:02 INFO - TEST-PASS | browser/extensions/doh-rollout/test/browser/browser_providerSteering.js | TRR URI set to auto-selected -
[task 2020-06-05T02:59:02.905Z] 02:59:02 INFO - Leaving test bound testProviderSteering
[task 2020-06-05T02:59:02.906Z] 02:59:02 INFO - Console message: [JavaScript Error: "WebExtension context not found!" {file: "resource://gre/modules/ExtensionParent.jsm" line: 994}]
[task 2020-06-05T02:59:02.906Z] 02:59:02 INFO - getContextById@resource://gre/modules/ExtensionParent.jsm:994:13
[task 2020-06-05T02:59:02.906Z] 02:59:02 INFO - recvAPICall@resource://gre/modules/ExtensionParent.jsm:849:24
[task 2020-06-05T02:59:02.906Z] 02:59:02 INFO - _recv@resource://gre/modules/ConduitsChild.jsm:78:20
[task 2020-06-05T02:59:02.906Z] 02:59:02 INFO - receiveMessage@resource://gre/modules/ConduitsParent.jsm:357:20
[task 2020-06-05T02:59:02.906Z] 02:59:02 INFO - JSActor query*_send@resource://gre/modules/ConduitsChild.jsm:63:11
[task 2020-06-05T02:59:02.906Z] 02:59:02 INFO - _send@resource://gre/modules/ConduitsChild.jsm:111:18
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - callParentAsyncFunction@resource://gre/modules/ExtensionChild.jsm:826:18
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - callAsyncFunction@resource://gre/modules/ExtensionChild.jsm:590:33
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - stub@resource://gre/modules/Schemas.jsm:2679:30
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - setSetting@moz-extension://967ee500-aabb-4aa6-93af-c6e7cbca7754/background.js:346:47
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - rememberTRRMode@moz-extension://967ee500-aabb-4aa6-93af-c6e7cbca7754/background.js:103:19
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - asyncsetState@moz-extension://967ee500-aabb-4aa6-93af-c6e7cbca7754/background.js:94:24
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - asyncheuristics@moz-extension://967ee500-aabb-4aa6-93af-c6e7cbca7754/background.js:283:26
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - asynconConnectionChanged@moz-extension://967ee500-aabb-4aa6-93af-c6e7cbca7754/background.js:547:19
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - AsyncapplySafeWithoutClone@resource://gre/modules/ExtensionCommon.jsm:614:24
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - fire@resource://gre/modules/ExtensionChild.jsm:775:37
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - recvRunListener@resource://gre/modules/ExtensionChild.jsm:779:13
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - _recv@resource://gre/modules/ConduitsChild.jsm:78:20
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - receiveMessage@resource://gre/modules/ConduitsChild.jsm:169:20
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - JSActor query*_send@resource://gre/modules/ConduitsChild.jsm:61:20
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - _send@resource://gre/modules/ConduitsParent.jsm:228:18
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - listener@resource://gre/modules/ExtensionParent.jsm:920:39
[task 2020-06-05T02:59:02.907Z] 02:59:02 INFO - applySafeWithoutClone@resource://gre/modules/ExtensionCommon.jsm:614:24
[task 2020-06-05T02:59:02.908Z] 02:59:02 INFO - applySafe@resource://gre/modules/ExtensionParent.jsm:502:17
[task 2020-06-05T02:59:02.908Z] 02:59:02 INFO - addListener/async/<@resource://gre/modules/ExtensionCommon.jsm:2411:39
[task 2020-06-05T02:59:02.908Z] 02:59:02 INFO - promise callbackasync@resource://gre/modules/ExtensionCommon.jsm:2409:34
[task 2020-06-05T02:59:02.908Z] 02:59:02 INFO - observerStatus@chrome://extensions/content/parent/ext-networkStatus.js:63:20
[task 2020-06-05T02:59:02.908Z] 02:59:02 INFO - simulateNetworkChange@chrome://mochitests/content/browser/browser/extensions/doh-rollout/test/browser/head.js:268:16
[task 2020-06-05T02:59:02.908Z] 02:59:02 INFO - testProviderSteering@chrome://mochitests/content/browser/browser/extensions/doh-rollout/test/browser/browser_providerSteering.js:68:3
[task 2020-06-05T02:59:02.908Z] 02:59:02 INFO - AsyncTester_execTest/<@chrome://mochikit/content/browser-test.js:1064:34
[task 2020-06-05T02:59:02.908Z] 02:59:02 INFO - async*Tester_execTest@chrome://mochikit/content/browser-test.js:1104:11
[task 2020-06-05T02:59:02.908Z] 02:59:02 INFO - nextTest/<@chrome://mochikit/content/browser-test.js:927:14
[task 2020-06-05T02:59:02.908Z] 02:59:02 INFO - SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:918:23
[task 2020-06-05T02:59:02.908Z] 02:59:02 INFO -
[task 2020-06-05T02:59:02.908Z] 02:59:02 INFO - Console message: [JavaScript Error: "NS_ERROR_NOT_AVAILABLE"]
[task 2020-06-05T02:59:02.908Z] 02:59:02 INFO - Buffered messages finished
[task 2020-06-05T02:59:02.909Z] 02:59:02 INFO - TEST-UNEXPECTED-FAIL | browser/extensions/doh-rollout/test/browser/browser_providerSteering.js | A promise chain failed to handle a rejection: [Exception... "Component is not available" nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)" location: "<unknown>" data: no] - stack: (No stack available.)
[task 2020-06-05T02:59:02.909Z] 02:59:02 INFO - Rejection date: Fri Jun 05 2020 02:59:02 GMT+0000 (Greenwich Mean Time) - false == true - JS frame :: resource://testing-common/PromiseTestUtils.jsm :: assertNoUncaughtRejections :: line 265
[task 2020-06-05T02:59:02.909Z] 02:59:02 INFO - Stack trace:
[task 2020-06-05T02:59:02.909Z] 02:59:02 INFO - resource://testing-common/PromiseTestUtils.jsm:assertNoUncaughtRejections:265
[task 2020-06-05T02:59:02.909Z] 02:59:02 INFO - chrome://mochikit/content/browser-test.js:nextTest:610
[task 2020-06-05T02:59:02.909Z] 02:59:02 INFO - chrome://mochikit/content/browser-test.js:testScope/test_finish/<:1464
[task 2020-06-05T02:59:02.909Z] 02:59:02 INFO - chrome://mochikit/content/browser-test.js:run:1379
[task 2020-06-05T02:59:02.909Z] 02:59:02 INFO - GECKO(10440) | MEMORY STAT | vsize 19406189MB | vsizeMaxContiguous 66966942MB | residentFast 1245MB
[task 2020-06-05T02:59:02.909Z] 02:59:02 INFO - TEST-OK | browser/extensions/doh-rollout/test/browser/browser_providerSteering.js | took 1708ms

Comment 13

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/integration/autoland/rev/60880b504f7335d9164561f77062af60476d3c37
https://hg.mozilla.org/integration/autoland/rev/286c0c0af787703eb89d593bc0345b4ee9c562cd
https://hg.mozilla.org/integration/autoland/rev/d55aac34140857cfed1c680cc9159fb666bd1cb3
https://hg.mozilla.org/integration/autoland/rev/95c0138bedc95f7b3788a540bf1e117984ec4545
https://hg.mozilla.org/mozilla-central/rev/60880b504f73
https://hg.mozilla.org/mozilla-central/rev/286c0c0af787
https://hg.mozilla.org/mozilla-central/rev/d55aac341408
https://hg.mozilla.org/mozilla-central/rev/95c0138bedc9

Comment 14

[(inactive) Nihanth Subramanya [:nhnt11]]

Comment on attachment 9150922 [details]
Bug 1631609 - Steer to network-indicated DoH endpoint if detected. r=valentin!,johannh!

Beta/Release Uplift Approval Request

• User impact if declined: This patchset enables automatic usage of provider-specific DoH endpoints and advances our efforts to ship DNS over HTTPS to more users.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: We are working with QA already and a test plan is being created.
• List of other uplifts needed: Bug 1640741
• Risk to taking this patch: Medium
• Why is the change risky/not risky? (and alternatives if risky): Not a trivial patch, but also not high-risk due to automated tests + QA support.
• String changes made/needed: None

Comment 15

[Julien Cristau [:jcristau]]

Per our conversation on slack please request data-review on the addition to the telemetry event.

Comment 16

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: Data review doc

Comment 17

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Comment on attachment 9156073 [details]
Data review doc

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way?

They are currently documented in https://github.com/mozilla/gecko-dev/blob/2f39179838c0ea407db05fcda1f8949e405490f4/browser/extensions/doh-rollout/experiments/heuristics/schema.json.

The heuristics will move out of the system-addon and the definition will move into Events.yaml in the next few cycles and will be documented in the probe dictionary.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the Firefox telemetry opt-out.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

Nhi Nguyen will monitor the telemetry.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, techical data.

5) Is the data collection request for default-on or default-off?

Default-on.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

no

7) Is the data collection covered by the existing Firefox privacy notice?

yes

8) Does there need to be a check-in in the future to determine whether to renew the data?

No, this is a permanent collection.

9) Does the data collection use a third-party collection tool?

No.

Comment 18

[Julien Cristau [:jcristau]]

Comment on attachment 9150922 [details]
Bug 1631609 - Steer to network-indicated DoH endpoint if detected. r=valentin!,johannh!

approved for 78.0b7

Comment 19

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/8d736d7eb44b
https://hg.mozilla.org/releases/mozilla-beta/rev/e74d135fa4a4
https://hg.mozilla.org/releases/mozilla-beta/rev/26abef4c3b16
https://hg.mozilla.org/releases/mozilla-beta/rev/9bd02f8e48f0