1521940 - collect pinning telemetry in release for mozilla sites that are essential for the operation of firefox

Status: RESOLVED FIXED

(Core :: Security: PSM, enhancement, P1)

Description

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

We collect pinning telemetry for some Mozilla web properties that have a functional impact on Firefox, but only for prelease populations. This should really be all populations:

https://telemetry.mozilla.org/probe-dictionary/?search=cert_pinning_moz_test_results_by_host&searchtype=in_name&channel=nightly
https://telemetry.mozilla.org/probe-dictionary/?search=cert_pinning_moz_results_by_host&searchtype=in_name&channel=nightly

Comment 1

[Julien Vehent]

For context, this is blocking bug 1494431 because we currently lack the necessary data to move forward. Since we need to collect release data, and it will take a while to ride there, can I ask to prioritize this change now-ish so we can measure impact when 66 hits the release channel?

Comment 2

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: bug 1521940 - collect pinning telemetry in release for mozilla sites that are essential for the operation of firefox data-review=chutten

The telemetry histograms CERT_PINNING_MOZ_TEST_RESULTS_BY_HOST and
CERT_PINNING_MOZ_RESULTS_BY_HOST collect information about whether or not key
pinning checks succeed in connections to Mozilla sites that affect the
functionality of Firefox. This patch changes these histograms so that we also
collect this data by default in release.

Comment 3

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Comment on attachment 9038379 [details]
bug 1521940 - collect pinning telemetry in release for mozilla sites that are essential for the operation of firefox data-review=chutten

Rob - let me know if I should ask someone else for this data review, but basically we're asking to expand two currently-existing telemetry probes to be opt-out rather than opt-in on release. They're category 1 and 2 data: part technical details (was the client able to build the right certificate chain for connections to specific Mozilla hosts) and part interaction (in some cases users would only connect to these hosts if they're using specific features).

Comment 4

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

(re comment 3)

Comment 5

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Comment on attachment 9038379 [details]
bug 1521940 - collect pinning telemetry in release for mozilla sites that are essential for the operation of firefox data-review=chutten

I've been told this needs to go through a full review at this point.

Comment 6

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: 1521940-pinning-telemetry-data-review

Comment 7

[Chris H-C :chutten]

Comment on attachment 9040858 [details]
1521940-pinning-telemetry-data-review

Are either of these histograms accumulated to when the user navigates to websites?

Comment 8

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Yes, if they visit the following:

addons.mozilla.org
addons.mozilla.net
aus4.mozilla.org
aus5.mozilla.org
firefox.com
accounts.firefox.com
api.accounts.firefox.com
sync.services.mozilla.com
cdn.mozilla.net
cdn.mozilla.org
download.mozilla.org
services.mozilla.com
telemetry.mozilla.org
testpilot.firefox.com
crash-reports.mozilla.com
crash-reports-xpsp2.mozilla.com
crash-stats.mozilla.com

Comment 9

[Julien Vehent]

Note that most of these are background services of Firefox. Users almost never "navigate" there, but all Firefox clients connect to them.

Comment 10

[Chris H-C :chutten]

Comment on attachment 9040858 [details]
1521940-pinning-telemetry-data-review

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions file ([Histograms.json](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Histograms.json)) and the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes. Dana Keeler is responsible for this collection.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 3, Web Activity.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Unclear.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview- due to collection of Category 3 data by default on release channel.

This collection may prove eligible for default-on collection in release channel if that the collection is limited to the list of properties in comment #8 is considered an adequate risk mitigation. (see https://wiki.mozilla.org/Firefox/Data_Collection#Eligibility_for_Default_on_Data_Collection)

ni?merwin for a decision: Is it okay that we are recording the number of (and some technical details about) connections (mostly infrastructure, but potentially identifiably web activity) to the mozilla-owned origins listed in comment #8?

Comment 11

[Merwin]

I think that is acceptable for the reason Julien mentioned - the user isn't navigating to these and therefore this isn't revealing of user web browsing. This is essentially interaction data because it reveals that the user interacted with a particular service integrated into Fx.

Comment 12

[Chris H-C :chutten]

Comment on attachment 9040858 [details]
1521940-pinning-telemetry-data-review

That's very clear guidance, thank you Marshall. I'll rerun the review.

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions file ([Histograms.json](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Histograms.json)) and the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes. Dana Keeler is responsible for this collection.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/906716408541
collect pinning telemetry in release for mozilla sites that are essential for the operation of firefox data-review=chutten r=chutten

Comment 14

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/906716408541