1436338 - Name resolve telemetry data collection for TRR

Status: RESOLVED FIXED

(Core :: Networking: DNS, defect, P3)

Description

[Daniel Stenberg [:bagder]]

Attachment: v4-0001-bug-1434852-introducing-TRR-DOH-r-valentin-mcmanu.patch

I'm working on bug 1434852, which adds an alternative name resolver functionality to Firefox. To properly evaluate how well and fast it runs, in particular in comparison to the existing "native resolver", I would like to add a few new telemetry probes.

All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.

1. What questions will you answer with this data?

How well the new resolver works. How fast it is, how often it has problems, how it compares to the old/existing one.

2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

Establish baselines and measure changes in product or platform quality or performance.

3. What alternative methods did you consider to answer these questions? Why were they not sufficient?

I can't think of any alternative ways that would be even close to be able to get similar coverage and accuracy.

4. Can current instrumentation answer these questions?

Not for the new proposed functionality, no.

5. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories on the found on the Mozilla wiki.

I'd like to add 6 new telemetry probes for this new functionality. I categorize them all as Category 1 “Technical data”.

6. How long will this data be collected? Choose one of the following:

I want to permanently monitor this data. (dstenberg)

7. What populations will you measure?

Everyone, in all channels.

8. If this data collection is default on, what is the opt-out mechanism for users?

nope

9. Please provide a general description of how you will analyze this data.

It will be used to determine how well the TRR code runs/works. It will be used to determine how DNS-over-HTTPS for users work in comparison to using native resolver.

10. Where do you intend to share the results of your analysis?

The analysis is meant to primary work as feedback to generate TRR improvements and to help us make further future decisions on how to do name resolving with Firefox and on the Internet as a whole. The data will appear in the regular telemetry web site.

Comment 1

[Chris H-C :chutten]

(rehoming to the component being measured)

Drive-by feedback: With a "high" of 60s, is it important that the new exponential measures (DNS_{TRR|NATIVE}_LOOKUP_TIME} match the old DNS_LOOKUP_TIME? Real-world values in DNS_LOOKUP_TIME have a 95th percentile around 1.2s (https://mzl.la/2s9QB00), so you could get some better resolution in the lower-value buckets of your new measures (where the differences between TRR and Native might lie) if you reduced the high bucket to 30s[1] or even 10s[2].

[1]: https://telemetry.mozilla.org/histogram-simulator/#low=1&high=30000&n_buckets=50&kind=exponential&generate=log-normal
[2]: https://telemetry.mozilla.org/histogram-simulator/#low=1&high=10000&n_buckets=50&kind=exponential&generate=log-normal

Comment 2

[François Marier [:francois]]

Comment on attachment 8948976 [details] [diff] [review]
v4-0001-bug-1434852-introducing-TRR-DOH-r-valentin-mcmanu.patch

Review of attachment 8948976 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Daniel Stenberg [:bagder] from comment #0)
> 8. If this data collection is default on, what is the opt-out mechanism for
> users?
> 
> nope

Note that there _is_ an opt-out mechanism for users: turning off telemetry from about:preferences.

Removing myself from r? since you'll need someone to review the code changes. Histograms.json looks fine. I will follow up with a data review.

Comment 3

[François Marier [:francois]]

datareview+

1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, in Histograms.json.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, telemetry setting.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?**

Yes, Daniel.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?  **

Category 1.

5) Is the data collection request for default-on or default-off?

Default on, all channels.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)**

No, permanent.

Comment 4

[Daniel Stenberg [:bagder]]

The code changes are already being reviewed in bug 1434852. The instructions for asking for a data collection review says I should attach a file with the changes, which is why I submitted the patch as I don't understand what other file I would offer and without a file attached, I couldn't r?...

Thanks!

Comment 5

[Chris H-C :chutten]

Typically I see people ask for Data Collection Review inside the bug implementing the data, which is why this flow might seem a little awkward.

Comment 6

[Daniel Stenberg [:bagder]]

Thanks, bug 1434852 has landed and with the datareview+ here I think this is done.