1522919 - Add telemetry for the fingerprinting and cryptomining content blocking categories

Status: RESOLVED FIXED

(Core :: Privacy: Anti-Tracking, enhancement, P3)

Description

[Steven Englehardt [:englehardt]]

To measure the following:

1. Whether or not fingerprinting / cryptomining blocking is enabled at startup
2. Whether or not fingerprinting / cryptomining blocking is enabled when "Report a problem" is clicked
3. Count of the number of page loads on which each category actively blocking at least one domain

We may want to consider doing this in a way that is more generic. I.e., send a ping that includes the set of active url-classifier features at startup, or have Report a problem's ping include all active blocklists.

Comment 1

[Johann Hofmann [:johannh]]

Please also add new event telemetry entries for the fp/cryptomining UI in the control center: https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/Events.yaml#677-688

Comment 2

[Johann Hofmann [:johannh]]

Just a note for whoever ends up picking this up:

(In reply to Steven Englehardt [:englehardt] from comment #0)

To measure the following:

1. Whether or not fingerprinting / cryptomining blocking is enabled at startup

We already have some mechanisms for the other content blocking features in https://searchfox.org/mozilla-central/rev/490ab7f9b84570573a49d7fa018673ce0d5ddf22/browser/components/nsBrowserGlue.js#1386

2. Whether or not fingerprinting / cryptomining blocking is enabled when "Report a problem" is clicked

This is already recorded automatically, if you give the feature a telemetryIdentifier (https://searchfox.org/mozilla-central/rev/e00ea598e52bbb35f8c45abf9c2eade17962bb5e/browser/base/content/browser-contentblocking.js#7) and add it to the "extra_keys" here: https://searchfox.org/mozilla-central/rev/490ab7f9b84570573a49d7fa018673ce0d5ddf22/toolkit/components/telemetry/Events.yaml#670-672,702-704

3. Count of the number of page loads on which each category actively blocking at least one domain

That's probably a new measurement, though we do have the "shield" histogram which is somewhat similar (https://searchfox.org/mozilla-central/rev/e00ea598e52bbb35f8c45abf9c2eade17962bb5e/browser/base/content/browser-contentblocking.js#816-820). Maybe we could make it a keyed histogram or something. A little creativity needed here.

Comment 3

[Erica Wright [:ewright]]

(In reply to Johann Hofmann [:johannh] from comment #2)

2. Whether or not fingerprinting / cryptomining blocking is enabled when "Report a problem" is clicked

This is already recorded automatically, if you give the feature a telemetryIdentifier (https://searchfox.org/mozilla-central/rev/e00ea598e52bbb35f8c45abf9c2eade17962bb5e/browser/base/content/browser-contentblocking.js#7) and add it to the "extra_keys" here: https://searchfox.org/mozilla-central/rev/490ab7f9b84570573a49d7fa018673ce0d5ddf22/toolkit/components/telemetry/Events.yaml#670-672,702-704

:englehardt, :johannh, You are correct that there is some telemetry already recorded here, but it is not whether the pref is enabled when "Report a problem" is clicked, it is whether a cryptominer/fingerprinter was being blocked when "Report a problem" is clicked. Which one do we want, or do we want both?

Comment 4

[Steven Englehardt [:englehardt]]

:englehardt, :johannh, You are correct that there is some telemetry already recorded here, but it is not whether the pref is enabled when "Report a problem" is clicked, it is whether a cryptominer/fingerprinter was being blocked when "Report a problem" is clicked. Which one do we want, or do we want both?

I meant whether a cryptominer/fingerprinter was being blocked when "Report a problem" is clicked, not whether the feature was simply enabled, but not blocking, on the current page. Thanks for catching that!

Comment 5

[Erica Wright [:ewright]]

Attachment: Bug 1522919 - Add basic telemetry for startup and control panel events for cryptomining and fingerprinting.

Include a boolean reporting whether these prefs are enabled on startup. inlcude extra information when user interacts with the control panel reporting if anything is being blocked.

Comment 6

[Erica Wright [:ewright]]

Attachment: data-review.txt

doc for data review please and thank you!

Comment 7

[Erica Wright [:ewright]]

3. Count of the number of page loads on which each category actively blocking at least one domain

Implementing this portion caused the discovery of a bug, it won't be able to be added until after bug 1528080 is fixed

Comment 8

[Chris H-C :chutten]

Comment on attachment 9044036 [details]
data-review.txt

DATA COLLECTION REVIEW RESPONSE

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate? 

Yes. This collection is Telemetry so is documented in its definitions files (Events.yaml and Scalars.yaml) and in the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off? 

Yes. This collection is Telemetry so can be controlled in Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Arthur Edelstein is responsible for the permanent monitoring of the scalars collecting whether prefs are enabled on startup. The other collections around the control panel are set to expire in Firefox 70.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

For the collection of whether the prefs are on, this is default-on for all channels. For the collection of the event information, this is default-on for pre-release channels only.

    Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

    Is the data collection covered by the existing Firefox privacy notice? 

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data? 

The events will expire in Firefox 70 so must be renewed or removed before then. The scalars are permanent.

---
Result: datareview+

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8ae2da51010e
Add basic telemetry for startup and control panel events for cryptomining and fingerprinting. r=johannh

Comment 10

[Cristina Coroiu [:ccoroiu]]

Sorry for the backout, landed again. Thanks

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/86a49f17f790
Add basic telemetry for startup and control panel events for cryptomining and fingerprinting. r=johannh on a CLOSED TREE

Comment 12

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/af0cd6ae2093
Backed out changeset 8ae2da51010e for mochitest failures on a CLOSED TREE

Comment 13

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/86a49f17f790

Comment 14

[Erica Wright [:ewright]]

Attachment: Bug 1522919 - Add histograms to count fingerprinting and cryptomining blocking states per page load.

Add histograms to count fingerprinting and cryptomining blocking states per page load.
Move shield histogram counts so it is not counting multiple per page load.

Comment 15

[Erica Wright [:ewright]]

Attachment: data-review-2.txt

Asking for data-review for part 2 of this patch. This adds a histogram which counts the number of page loads, and for each page load how many pages block or allow a fingerprinter/ cryptominer.

Comment 16

[Chris H-C :chutten]

Comment on attachment 9048236 [details]
data-review-2.txt

Preliminary note:

The data collection review documents that you intend that these be collected in all channels, but the Histograms.json definitions will only be recorded in pre-release channels due to them not having the correct `releaseChannelCollection` value. I will assume you do wish to record in all channels for the purposes of this review (so that if you do change the definition you need not refile the review).

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions file [Histograms.json](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Histograms.json) and the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Arthur Edelstein is responsible.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/39c2628de53e
Add histograms to count fingerprinting and cryptomining blocking states per page load. r=johannh

Comment 18

[Cristina Coroiu [:ccoroiu]]

Backed out changeset 39c2628de53e (bug 1522919) for browser chrome failures at content/test/trackingUI/browser_trackingUI_cryptominers.js

Backout: https://hg.mozilla.org/integration/autoland/rev/783538d74b6ab0dbffd89ac71eda0d8952394037

Failure push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=39c2628de53e5a87b23737ae0364705197cef9b1&selectedJob=232210625

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=232210625&repo=autoland&lineNumber=4143

[task 2019-03-06T17:52:44.972Z] 17:52:44 INFO - TEST-PASS | browser/base/content/test/trackingUI/browser_trackingUI_cryptominers.js | Shows an exception when appropriate -
[task 2019-03-06T17:52:44.974Z] 17:52:44 INFO - Buffered messages finished
[task 2019-03-06T17:52:44.975Z] 17:52:44 INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/trackingUI/browser_trackingUI_cryptominers.js | The correct number of page loads have been recorded - 3 == 1 - JS frame :: chrome://mochitests/content/browser/browser/base/content/test/trackingUI/browser_trackingUI_cryptominers.js :: testTelemetry :: line 119
[task 2019-03-06T17:52:44.976Z] 17:52:44 INFO - Stack trace:
[task 2019-03-06T17:52:44.977Z] 17:52:44 INFO - chrome://mochitests/content/browser/browser/base/content/test/trackingUI/browser_trackingUI_cryptominers.js:testTelemetry:119
[task 2019-03-06T17:52:44.977Z] 17:52:44 INFO - chrome://mochitests/content/browser/browser/base/content/test/trackingUI/browser_trackingUI_cryptominers.js:testIdentityState:58
[task 2019-03-06T17:52:44.977Z] 17:52:44 INFO - chrome://mochitests/content/browser/browser/base/content/test/trackingUI/browser_trackingUI_cryptominers.js:test:128
[task 2019-03-06T17:52:44.977Z] 17:52:44 INFO - chrome://mochikit/content/browser-test.js:Tester_execTest/<:1106
[task 2019-03-06T17:52:44.977Z] 17:52:44 INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1134
[task 2019-03-06T17:52:44.977Z] 17:52:44 INFO - chrome://mochikit/content/browser-test.js:nextTest/<:995
[task 2019-03-06T17:52:44.977Z] 17:52:44 INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<:803

Comment 19

[Erica Wright [:ewright]]

Thank you, I'm looking into it

Comment 20

[Erica Wright [:ewright]]

Could you dig into this on linux for me?

Comment 21

[Johann Hofmann [:johannh]]

We chatted about it :)

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b651bbce8a99
Add histograms to count fingerprinting and cryptomining blocking states per page load. r=johannh

Comment 23

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/b651bbce8a99