Closed Bug 1484251 Opened Last year Closed Last year

Telemetry for Content Blocking

Categories

(Firefox :: Protections UI, enhancement, P1)

enhancement

Tracking

()

RESOLVED FIXED
Firefox 64
Tracking Status
firefox64 --- fixed

People

(Reporter: johannh, Assigned: johannh)

References

(Blocks 1 open bug)

Details

(Whiteboard: [privacy-panel])

Attachments

(5 files)

We'd like to add telemetry for the new Content Blocking feature. pdol made a wish list here:

https://docs.google.com/document/d/1uUxoAh718myve0-DzVPRstYgMPsMdlcK1n_K05ke7J4/edit

Talking to telemetry people these goals seem pretty reasonable to attain.
Duplicate of this bug: 1482285
Whiteboard: [privacy-panel-64]
Priority: P2 → P1
Target Milestone: --- → Firefox 64
Assignee: nobody → jhofmann
Status: NEW → ASSIGNED
This probe was always supposed to measure TP enabled status for a session, not a window.
This removes the old TRACKING_PROTECTION_EVENTS probe and replaces it with new
Telemetry events that record basic user interaction in the identity popup.

We are now measuring interaction with more elements than before, not just block/unblock.

We're also dropping the old way of measuring updates to onSecurityChange,
since with the recent changes to content blocking it would just record most page loads.
Peter, Tanvi (and everyone else), can you please take a look at the Event measurements introduced in part 2 and part 3 and also check that the change we're making in part 1 (recording feature usage at startup instead of window open) is fine with you?

Thanks!
Flags: needinfo?(tanvi)
Flags: needinfo?(pdolanjski)
Attached file request.md
Attachment #9012124 - Flags: review?(francois)
Blocks: 1475670
Attachment #9012116 - Attachment description: Bug 1484251 - Part 1 - Record Tracking Protection enabled at startup, not at window open. r=Gijs → Bug 1484251 - Part 1 - Record Tracking Protection enabled at startup, not at window open. r=paolo
Attachment #9012117 - Attachment description: Bug 1484251 - Part 2 - Record telemetry for Content Blocking, FastBlock and Cookie Blocking status at startup. r=Gijs → Bug 1484251 - Part 2 - Record telemetry for Content Blocking, FastBlock and Cookie Blocking status at startup. r=paolo
Attachment #9012118 - Attachment description: Bug 1484251 - Part 3 - Use Telemetry events to record interactions in the identity popup. r=Gijs → Bug 1484251 - Part 3 - Use Telemetry events to record interactions in the identity popup. r=paolo
Attachment #9012119 - Attachment description: Bug 1484251 - Part 4 - Test updates for new content blocking telemetry. r=Gijs → Bug 1484251 - Part 4 - Test updates for new content blocking telemetry. r=paolo
Comment on attachment 9012116 [details]
Bug 1484251 - Part 1 - Record Tracking Protection enabled at startup, not at window open. r=Gijs

:Gijs (out Thu 27 - Sun 30 / 9;  he/him) has approved the revision.
Attachment #9012116 - Flags: review+
Chris, do you still like to review all new Event probes?
Flags: needinfo?(chutten)
Comment on attachment 9012124 [details]
request.md

1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, in Histograms.json, Scalars.yaml and Events.yaml.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, telemetry setting.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?**

Presumably pdol.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?  **

Category 2.

5) Is the data collection request for default-on or default-off?

Default ON for some of it, on all channels.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

No, telemetry alerts are sufficient.
Attachment #9012124 - Flags: review?(francois) → review+
(In reply to Johann Hofmann [:johannh] from comment #6)
> Peter, Tanvi (and everyone else), can you please take a look at the Event
> measurements introduced in part 2 and part 3 and also check that the change
> we're making in part 1 (recording feature usage at startup instead of window
> open) is fine with you?
> 
> Thanks!

This looks okay from my perspective.
Flags: needinfo?(pdolanjski)
(In reply to François Marier [:francois] from comment #9)
> Chris, do you still like to review all new Event probes?

The important thing is checking that the volume of these events will be small enough, if they're always enabled. In this case they're tied to user interactions, which makes them likely to be low-volume.

Thanks for the heads-up!
Flags: needinfo?(chutten)
Looping in Sunah for any feedback she might have on taxonomy and event use.
Flags: needinfo?(ssuh)
In addition to the review comments, if you need to answer questions that require the data to be combined, you may want to write code that does that beforehand and submits also the combined flag, to simplify analysis later.

One example of these questions from the document might be "Of those who have hit Add Blocking, what percentage actually have Tracking Protection set to Always?"
In terms of volume since this is pre-release only I don't think there's going to be any issues with continual recording, though I think making enable/disable of the event category a pref seems to be emerging as a best practice and would be useful here in case we do end up wanting to turn it off out-of-band.

Regarding taxonomy, if adding dismissal without an action is possible it'd probably make sense to add that as well.
Flags: needinfo?(ssuh)
(In reply to :Paolo Amadini from comment #14)
> In addition to the review comments, if you need to answer questions that
> require the data to be combined, you may want to write code that does that
> beforehand and submits also the combined flag, to simplify analysis later.
> 
> One example of these questions from the document might be "Of those who have
> hit Add Blocking, what percentage actually have Tracking Protection set to
> Always?"

Yes, talking to chutten it seems like it's a good idea to associate the event data with the current state of content blocking as extra fields. Thanks for raising that point.

(In reply to Sunah Suh (she/her) [:sunahsuh] from comment #15)
> In terms of volume since this is pre-release only I don't think there's
> going to be any issues with continual recording, though I think making
> enable/disable of the event category a pref seems to be emerging as a best
> practice and would be useful here in case we do end up wanting to turn it
> off out-of-band.

I'll add a pref, thanks!
 
> Regarding taxonomy, if adding dismissal without an action is possible it'd
> probably make sense to add that as well.

Sorry, I didn't really understand this, do you mean dismissal of the identity popup without interaction? It's an interesting idea but not possible without recording some intermediate value and this bug has already grown large enough that I'd prefer to defer this to a follow-up.
Flags: needinfo?(tanvi)
Attachment #9012116 - Attachment description: Bug 1484251 - Part 1 - Record Tracking Protection enabled at startup, not at window open. r=paolo → Bug 1484251 - Part 1 - Record Tracking Protection enabled at startup, not at window open. r=Gijs
Attachment #9012117 - Attachment description: Bug 1484251 - Part 2 - Record telemetry for Content Blocking, FastBlock and Cookie Blocking status at startup. r=paolo → Bug 1484251 - Part 2 - Record telemetry for Content Blocking, FastBlock and Cookie Blocking status at startup. r=paolo,chutten
Attachment #9012118 - Attachment description: Bug 1484251 - Part 3 - Use Telemetry events to record interactions in the identity popup. r=paolo → Bug 1484251 - Part 3 - Use Telemetry events to record interactions in the identity popup. r=paolo,chutten
Whiteboard: [privacy-panel-64] → [privacy-panel]
Full TP would be #3 of all addons next week or so. What a success! https://data.firefox.com/dashboard/usage-behavior
Do you already have some Beta 63 numbers about the same growth? Has it been regressed e.g. by bug 1476224 or can it ride into release because growth numbers are the same or better? For the worst case, would you be able to show the Content Blocking panel, but keeping the easy to understand Full TP main menu toggle? (bug 1490388 comment 1) Thanks!
Attachment #9012117 - Attachment description: Bug 1484251 - Part 2 - Record telemetry for Content Blocking, FastBlock and Cookie Blocking status at startup. r=paolo,chutten → Bug 1484251 - Part 2 - Record telemetry for Content Blocking, FastBlock and Cookie Blocking status at startup. r=Gijs,chutten
Attachment #9012118 - Attachment description: Bug 1484251 - Part 3 - Use Telemetry events to record interactions in the identity popup. r=paolo,chutten → Bug 1484251 - Part 3 - Use Telemetry events to record interactions in the identity popup. r=Gijs,chutten
Attachment #9012119 - Attachment description: Bug 1484251 - Part 4 - Test updates for new content blocking telemetry. r=paolo → Bug 1484251 - Part 4 - Test updates for new content blocking telemetry. r=Gijs
Pushed by jhofmann@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8297dbe74765
Part 1 - Record Tracking Protection enabled at startup, not at window open. r=Gijs
https://hg.mozilla.org/integration/autoland/rev/734669014545
Part 2 - Record telemetry for Content Blocking, FastBlock and Cookie Blocking status at startup. r=Gijs
https://hg.mozilla.org/integration/autoland/rev/8794df889e3f
Part 3 - Use Telemetry events to record interactions in the identity popup. r=Gijs
https://hg.mozilla.org/integration/autoland/rev/29d9b2d2192d
Part 4 - Test updates for new content blocking telemetry. r=Gijs
Depends on: 1522794
Regressions: 1564241
You need to log in before you can comment on or make changes to this bug.