1553700 - DNS over HTTPS Heuristics Measurement Add-on

Status: RESOLVED FIXED

(Firefox :: General, task, P1)

Description

[Wayne Thayer]

A new add-on is required to implement the DoH exclusion heuristics for DNS filtering (e.g. redirect from www.google.com to safesearch.google.com) and split horizon [1].

This add-on will be used for an experiment to gather measurements on these heuristics [2].

[1] https://docs.google.com/document/d/1P8iMpSQXfzJ5CaXZNNAE9QE3YJBdvPw1gHd7mCTBFsA/edit?ts=5cde7e0b
[2] https://experimenter.services.mozilla.com/experiments/dns-over-https-experiment-7-gather-statistics-on-the-effectiveness-of-heuristics/ and https://bugzilla.mozilla.org/show_bug.cgi?id=1552783

Comment 1

[Austin Hounsel [:ahounsel]]

I've attached the build to be signed and sent to QA for testing.

Comment 2

[Austin Hounsel [:ahounsel]]

Attachment: Build of the addon to be signed and sent to QA for testing

Comment 3

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 4

[Austin Hounsel [:ahounsel]]

Attachment: Review request for data collection

Comment 5

[Austin Hounsel [:ahounsel]]

Attachment: Code-reviewed build of the addon that needs to be signed for QA

Comment 6

[Michael Cooper [:mythmon]]

Attachment: testing

Comment 7

[Ben Miroglio [:bmiroglio]]

Comment on attachment 9077567 [details]
Review request for data collection

Hi Austin

Is there a reason the two existing probes listed below are not sufficient in place of your proposed `uniqueDomains` metric. If not, could you be more specific about what `uniqueDomains` is measuring?

https://telemetry.mozilla.org/probe-dictionary/?search=browser.engagement&detailView=scalar%2Fbrowser.engagement.unique_domains_count
https://telemetry.mozilla.org/probe-dictionary/?search=browser.engagement&detailView=scalar%2Fbrowser.engagement.total_uri_count

Comment 8

[Austin Hounsel [:ahounsel]]

(In reply to Ben Miroglio [:bmiroglio] from comment #7)

Comment on attachment 9077567 [details]
Review request for data collection

Hi Austin

Is there a reason the two existing probes listed below are not sufficient in
place of your proposed uniqueDomains metric. If not, could you be more
specific about what uniqueDomains is measuring?

https://telemetry.mozilla.org/probe-dictionary/?search=browser.
engagement&detailView=scalar%2Fbrowser.engagement.unique_domains_count
https://telemetry.mozilla.org/probe-dictionary/?search=browser.
engagement&detailView=scalar%2Fbrowser.engagement.total_uri_count

The browser.engagement.unique_domains_count metric does not suffice because it only counts "test.example.com" and "other.example.com" once. For our study, we would count those domains twice.

Similarly, the browser.engagement.total_uri_count metric does not suffice because it counts each URI the user is visiting, e.g. "example.com/foo.html" and "example.com/bar.html". For our study, we would only count these pages once because they share the same domain name.

In short, uniqueDomains counts the total number of unique fully-qualified domain names a user has visited in a browsing session. We are interested understanding how many unique fully-qualified domain names a user visits, such as "test.example.com" and "other.example.com". One domain might be under parental control while the other might not. Similarly, one domain might be resolved in a unique way by an enterprise DNS setup while the other might not.

Let me know if you have any other questions!

Comment 9

[Austin Hounsel [:ahounsel]]

Attachment: Code-reviewed build of the addon to be signed for QA

Comment 10

[Ben Miroglio [:bmiroglio]]

Comment on attachment 9077567 [details]
Review request for data collection

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, this bug is public and the schema is described here: https://gist.github.com/ahounsel/2c07211304fc903040bac3e6cb5f51ca. Since this is an experiment, no permanent document exists.

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection deployed through Normandy and users can opt-out of such studies in the Firefox Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

No. This collection will expire 07-23-2019.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

No, this is part of an experiment, and affects ~0.5% of Firefox 68 Release users in the US.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

Yes.  ahounsel is responsible for renewing or removing the collection before it expires on 07-23-2019.

---
Result: datareview+

Comment 11

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 12

[Jonathan Kingston [:jkt] he/him]

Attachment: detection_logic_for_dns-over-https-1.0.1.zip

Can you also sign this too, it fixes: https://github.com/jonathanKingston/http-dns/issues/43

Comment 13

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 14

[Austin Hounsel [:ahounsel]]

Michael, we have passed QA. Could you sign version 1.0.1 of the addon again for Release? It's available in the above attachments, and at this link: https://bugzilla.mozilla.org/attachment.cgi?id=9078176

Comment 15

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 16

[Austin Hounsel [:ahounsel]]

Attachment: Unsigned addon, ready for release

Comment 17

[Michael Cooper [:mythmon]]

This most recent add-on is different than the previous 1.0.1 version because it doesn't have the testing flag set. This is a change to the add-on and so needs to carry a different version number. Additionally, since it is a changed add-on, QA will need to sign off on this version. QA doesn't need to re-test the add-on if they don't deem it needed, but they need to sign off on the new version being ready to ship.

Comment 18

[Austin Hounsel [:ahounsel]]

Attachment: Final addon to be re-signed by QA; "testing" flag set to false

This addon needs to be re-signed by QA because I forgot to set the "testing" flag to false for sending Telemetry data. That is the only change we made apart from updating the version number. According to Michael (:mythmon), it could be signed-off without testing if QA deems the change small enough.

Comment 19

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 20

[Austin Hounsel [:ahounsel]]

Version 1.0.2 of the addon has been reviewed by QA, and they gave us the green light. Michael, could you sign the addon for release at your earliest convenience? We are planning on launching the experiment tomorrow, pending VP sign-off.

https://bugzilla.mozilla.org/attachment.cgi?id=9078497

Comment 21

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 22

[Austin Hounsel [:ahounsel]]

Attachment: Found a bug before release. Need to send back to QA.

Comment 23

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 24

[Austin Hounsel [:ahounsel]]

Attachment: Minor bug fix for QA

Comment 25

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 26

[Austin Hounsel [:ahounsel]]

Addon has gone through QA again. We are ready to sign 1.0.4 for release.
https://bugzilla.mozilla.org/attachment.cgi?id=9079105

Comment 27

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 28

[Austin Hounsel [:ahounsel]]

Attachment: Re-starting QA process

Comment 29

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 30

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Comment on attachment 9077567 [details]
Review request for data collection

The proposed collection has changed, so we should put this up for review again.

Comment 31

[Austin Hounsel [:ahounsel]]

Attachment: Updated review request for data collection

Comment 32

[Austin Hounsel [:ahounsel]]

Attachment: Fixed bug identified by QA

Comment 33

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 34

[Megan McCorquodale]

Comment on attachment 9080797 [details]
Updated review request for data collection

DATA COLLECTION REVIEW RESPONSE:

1.    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, this bug is public and the schema is described here: https://gist.github.com/ahounsel/2c07211304fc903040bac3e6cb5f51ca. Since this is an experiment, no permanent document exists.

2.    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection deployed through Normandy and users can opt-out of such studies in the Firefox Preferences.

3.    If the request is for permanent data collection, is there someone who will monitor the data over time?

No. This collection will expire 08-06-2019.

4.    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction data.

5.    Is the data collection request for default-on or default-off?

Default Off. This is part of an experiment, and affects ~0.5% of Firefox 68 Release users in the US.

6.    Does the instrumentation include the addition of any new identifiers?

No new identifiers.

7.    Is the data collection covered by the existing Firefox privacy notice?

Yes.

8.    Does there need to be a check-in in the future to determine whether to renew the data?

Yes.  ahounsel is responsible for renewing or removing the collection before it expires on 08-06-2019.

---
Result: datareview+

Comment 35

[Austin Hounsel [:ahounsel]]

We got approval from QA again. We are ready to sign version 1.0.6 for release.

Comment 36

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 37

[Austin Hounsel [:ahounsel]]

The study successfully ended on August 13th, 2019, and the results have been analyzed. I'm closing this bug because there is no work left to be done on the addon.