1231376 - [experiment] Audit telemetry probes for Tracking Protection

Status: NEW

(Firefox :: Settings UI, defect, P3)

Description

[Panos Astithas (he/him) [:past] (please ni?)]

We want to add instrumentation for all the new settings: users running TP in PBM, always, not at all, users who change a blocklist. We should already have instrumentation around users running DNT, but let's verify this.

Comment 1

[François Marier [:francois]]

The existing telemetry we have is displayed here: https://people.mozilla.org/~fmarier/tracking-dashboard/

Comment 2

[Javaun Moradi [:javaun]]

Paolo and I spoke about how we already had a lot of telemetry. This bug is just for rigor to audit what we have for gaps and male sure it meets our needs, since this data will be critical to launch. Ilana is a great person to help us here,  she's assisted before and I've already spoke with Matt G

Comment 3

[:Paolo Amadini]

I've taken a look at the current histograms.

We have telemetry for whether Tracking Protection is globally enabled in private windows through the preferences, and separately for non-private windows as well.

This data is gathered at every page load. Using this denominator means that results will be skewed towards users who visit lots of pages, or refresh the same page many times (like a pinned tab). There is no right or wrong denominator, for example gathering for each window opened or at each startup would skew results towards users who open a lot of windows or restart the browser more than others. An improvement here would could just be to be explicit about what we're measuring in the histogram description.

We also measure events about disable/enable button usage. This can tell us whether one button is clicked more often than the other one, and we have that data. It's difficult to read because in the histogram we're collecting something in bucket 0 that is not very useful:
 
https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-01-01&keys=__none__!__none__!__none__&max_channel_version=nightly%252F46&measure=TRACKING_PROTECTION_EVENTS&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2015-12-14&table=0&trim=1&use_submission_date=0

It's the number of onSecurityChange events, and not what the histogram description says. Since I don't know whether to number of times the Control Center is opened is useful to know, to fix this hisotgram it may be enough to remove this line:

http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser-trackingprotection.js#133

We can just ignore bucket 0 when looking at data for button usage.

We also have telemetry about the shield appearing or not during a page load, or appearing with a strikethrough, which seems correctly implemented to me.

We're not gathering histograms for switching to non-default list. We could track known values of the "urlclassifier.trackingTable" preference, though again we have to choose an appropriate denominator.

Comment 4

[Javaun Moradi [:javaun]]

Ilana, can you chime in on the denominator question above? Also any other input on how to proceed with the counting Paolo mentions.

To requirements, I'll add:
* We want to which list the user is using (default or strict list). 

* We could also measure the number of site-exceptions a user has stored. Not the actual pages, that would be a privacy link

Comment 6

[Ilana]

The denominator is really a matter of preference. Personally, I'd grab both and calculate two quotients - the one with page loads is better for seeing overall breakage, whereas "per startup" is closer to clientID, and will measure pain/person. Is there any reason not to do this by clientID as well?

To measure preference changes, I think the most useful thing you're looking for is #disables/enabled user. Thus, I'd count an active change of disables with the number of enabled periods for a user (thus: disabling and then reenabling is two enabled periods) in the denominator (note: this will skew towards more disables if users disable a large number of times, but that can be circumvented by converting to a "did change to disabled at least once in a time period" by clientID).