1577030 - Telemetry to know # of trackers blocked

Status: RESOLVED FIXED

(Firefox :: Protections UI, enhancement, P1)

Description

[Erica Wright [:ewright]]

We need to collect Telemetry numbers for an estimated average and aggregate number of trackers blocked for marketing purposes.

We can collect the number of trackers blocked per day per person.

Information we need from marketing:

• Do we need to collect this information on release channels?
• How will we be displaying this information? ex. "X trackers blocked for all users in the past month", "X trackers blocked per person per day"
• For average numbers do we want only an active user, or should we somehow count inactive users?

Comment 1

[chsiang]

My feedback on the 3 questions. Also included Liz from marketing to weigh in in case I miss anything.

1. Do we need to collect this information on release channels? Yes

2. How will we be displaying this information? ex. "X trackers blocked for all users in the past month", "X trackers blocked per person per day"
   Those two and a bit more context on where we might use those those numbers.

"X trackers blocked for all users in the past month":
There is a plan to build a website showing stats of our privacy related features. For instance, for monitor, we can show " x breach alert emails sent last months"
And this information can be shown on the website for users to see how this number changes over time.

"X trackers blocked per person per day":
This can be a CTA when we use it on social or over email communication to encourage people to start using Firefox as their main browser cuz this x number of trackers is the threat all users face everyday.

3. For average numbers do we want only an active user, or should we somehow count inactive users?
   We should count only an active user unless it technically too challenging.

Comment 2

[Johann Hofmann [:johannh]]

Based on this I think the best we can do is a scalar count of "how many trackers were blocked for this user yesterday"?

For average numbers do we want only an active user, or should we somehow count inactive users?

We will only get submissions from "active" users (users that open their Firefox frequently enough to send telemetry).

Comment 3

[Liz Hull]

Hi! Yes, thanks Cindy this is great.

Just to confirm, the main two messages we'd love to be able to use are:

1. Here's how many trackers have been blocked in total (globally, or across all active users of Firefox) in a given time frame.
   --Ideally we'd love to be able to show the number of total trackers blocked in a given day on a specific week. For launch week especially, we'd love to consistently update people about how the number is growing to entice them to join Firefox and start blocking those trackers for themselves. We're hoping to set a goal and slowly fill in the progress like an old-school fundraising thermometer until we hit that goal. From there on, monthly would be fine.
2. Here's how many trackers on average individuals block per day.

Comment 4

[Johann Hofmann [:johannh]]

For launch week especially, we'd love to consistently update people about how the number is growing to entice them to join Firefox and start blocking those trackers for themselves.

Chris, does our telemetry pipeline support updating the data fast enough for this to become a thing? Assuming we would make a scalar that reports how many trackers have been blocked for an invidiual user on the previous day. I assume we can accumulate all the scalar values via a query in STMO?

Thanks!

Comment 5

[Chris H-C :chutten]

I'm not sure what the latency on main_summary is in the new BigQuery world, but I do know we have the capability for near-realtime analysis in a couple places in the stack.

There is a latency wrinkle about "main" pings, however. They're sent at the end of sessions or about daily, so there's reporting delay there even if we wanted to go with a pseudo-realtime analysis. But a Scalar counting them up is probably the best choice for how to measure this in Firefox.

Lemme ni?mreid for a "What do we have that could sum a specific 'main' ping Scalar in near-realtime" inquiry.

Comment 6

[Mark Reid [:mreid]]

As Chris said, incoming scalar data is available for analysis within a few minutes, which should be plenty timely to show population aggregates. The latency of when clients send this data is a bigger factor here, since clients accumulate the counters for some time (~a day) before sending the data back.

We can easily add daily aggregates of this scalar to the existing clients_daily dataset which would give per-client info up to yesterday.

Comment 7

[Erica Wright [:ewright]]

Attachment: Bug 1577030 - Telemetry to track total numbers of trackers blocked.

Comment 8

[Erica Wright [:ewright]]

Attachment: request for data-review

Comment 9

[Chris H-C :chutten]

Comment on attachment 9090168 [details]
request for data-review

Hm. The embedded google doc isn't visible by me. Could you copy and paste the data review request into an attachment as plain text?

Comment 10

[Erica Wright [:ewright]]

Attachment: data-review request

Comment 11

[Chris H-C :chutten]

Comment on attachment 9090454 [details]
data-review request

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions file [Scalars.yaml](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Scalars.yaml) and the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, :chsiang is responsible.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

Yes. :chsiang is responsible for renewing or removing the collection before it expires in Firefox 75.

---
Result: datareview+

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2d56b9e09bd3
Telemetry to track total numbers of trackers blocked. r=johannh

Comment 13

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/2d56b9e09bd3

Comment 14

[Erica Wright [:ewright]]

Comment on attachment 9090137 [details]
Bug 1577030 - Telemetry to track total numbers of trackers blocked.

Beta/Release Uplift Approval Request

• User impact if declined: This telemetry is collecting data on the number of trackers blocked. We'd like to be able to act on this data as soon as possible. This will be in conjunction with Skyline efforts.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): patch containing only telemetry, no ui nor interaction changes.
• String changes made/needed: none

Comment 15

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 9090137 [details]
Bug 1577030 - Telemetry to track total numbers of trackers blocked.

Add telemetry in support of Skyline projects/tracking protection.
We're still early in beta so we have time to catch potential problems here.
Has data review, OK for uplift for beta 6.

Comment 16

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/releases/mozilla-beta/rev/6fb937a562b9