1557050 - Telemetry for protection report

Status: RESOLVED FIXED

(Firefox :: Protections UI, enhancement, P1)

Description

[Erica Wright [:ewright]]

As the product team, we’d like to know the product is used, so we can improve it.

• Put a pref for users who have opened protection report
• Know the path that leads to protection report (privacy UI, hamburger menu)
• Know how often users revisit the report after they first visit it

Comment 1

[Johann Hofmann [:johannh]]

Cindy, can you please share the document with our expectations for Telemetry for protection report. Also, would you mind filing a data review request?

Thank you!

Comment 2

[Johann Hofmann [:johannh]]

We should also send the correct funnel ids for links outbound.

Comment 3

[Leif Oines [:loines]]

put together an initial list of events here https://docs.google.com/spreadsheets/d/1DeaYrCg9TdhZe2HOWFvKebRQvyWeeo2IlWypj6UPisE/edit?usp=sharing (fixed sharing on the link)

Comment 4

[chsiang]

Request for data review form. Will flag a data steward once this document is complete.
https://docs.google.com/document/d/1tgvT6v-JTIYibJeDTGEyiBlCf8SMYA3hI2VpPEETQ0k/edit

Comment 5

[Erica Wright [:ewright]]

Attachment: Bug 1557050 - Add basic telemetry to protection report.

Comment 6

[Leif Oines [:loines]]

Attachment: Data collection review for data steward

Data review request, Tim feel free to hand off to someone else

Comment 7

[Leif Oines [:loines]]

Comment on attachment 9081681 [details]
Data collection review for data steward

per erica's request I'm removing some of the events (the proxy event, the doorhanger events and the "learn more" link on the tracking report
) from the data review as some of the features aren't implemented yet or the events will be instrumented as part of another bug/work.

>Request for data collection review form
>All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.
>
>Tracking bug 1557050 
>What questions will you answer with this data?
>How frequently users interact with the Protection Report and what buttons they are clicking within the menu.
>
>Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:
>Mozilla needs to answer these questions because it will help address product requirements. More specifically, it will allow us to answer how many people interact with the Protection Report and in what frequency. It will also allow us to know how many users opt in to Monitor, Lockwise, and Secure Proxy.
>
>What alternative methods did you consider to answer these questions? Why were they not sufficient?
>No alternative methods were considered because there is no other way of gathering this information except through telemetry.
>
>Can current instrumentation answer these questions?
>No, this is a new feature and it does not yet have any telemetry associated with it.
>
>List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories on the Mozilla wiki
>Leifâs Note: some of the links that are included in the protection report have an `entrypoint` parameter attached to them, so that the landing page can track that the user came from the protection report. 
>Telemetry spec was based on these designs: https://mozilla.invisionapp.com/share/RARKT5F7KDH#/screens/360923276
>
>Event for when the protection report is opened (Category 2)
>Protection_report.protection_report (method = show)
>
>Event for when the protection report is closed (Category 2)
>Protection_report.protection_report (method = close)
>
>Event for when the user when user clicks the link to lockwise mobile (Category 2)  Protection_report.lockwise_app_link 
>
>Event for when the user when user clicks the button to open the pw manager (Category 2)  Protection_report.lockwise_open_button 
>
>Event for when the user when user clicks the button to turn on sync (Category 2)  Protection_report.lockwise_sync_link 
>
>Event for when the user when user clicks the lockwise âhow it works linkâ (Category 2)  Protection_report.lockwise_about_link 
>
>Event for when the user when user clicks the button to open lockwise *from the monitor pane*  (Category 2)  Protection_report.lockwise_open_breach_link 
>
>Event for when the user when user clicks the button to open the monitor website to view a breach report  (Category 2)  Protection_report.monitor_report_link 
>
>Event for when the user when user clicks the âhow it worksâ button for montior  (Category 2)  Protection_report.monitor_about_link 
>
>Event for when the user when user clicks the âsign up for monitorâ button for montior  (Category 2)  Protection_report.monitor_signup_button 
>
>Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.
>
>How long will this data be collected? Choose one of the following:
>I want to monitor this data until version 75, then maybe renew. 
>Contact: [e-mail],
>
>What populations will you measure?
>Which release channels? All
>Which countries? All
>Which locales? All
>No other filters
>
>If this data collection is default on, what is the opt-out mechanism for users?
>User can opt out of this telemetry by turning telemetry off generally.
>
>Please provide a general description of how you will analyze this data.
>We plan to analyze this data to know how frequently users are opening the tracking report and making use of its features. We also plan to use views of the tracking report as a top of funnel number in determining how many users sign up for monitor or begin to use Lockwise. In the future these events may be used as dependent measures in experiments related to the tracking report, but those are not scoped as of yet. 
>
>Where do you intend to share the results of your analysis?
>Only with Mozilla employees/NDA mozillians
>
>Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? 
>Not currently, but events may eventually be sent to amplitude in the medium-to-long term future.
>

Comment 8

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Comment on attachment 9081681 [details]
Data collection review for data steward

Thanks!

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, in Events.yaml.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the Firefox telemetry opt-out.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

n/a

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, interaction data.

5) Is the data collection request for default-on or default-off?

Default-on.

6) Does the instrumentation include the addition of **any *new* identifiers**?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No)

Yes, Leif is responsible for renewing the collections before they expire in 75.

9) Does the data collection use a third-party collection tool? **If yes, escalate to legal.**

No.

Comment 9

[Erica Wright [:ewright]]

:tdsmith :loines some of the names were too long, and un-allowed in the Yaml file, please note that I shortened them.
"lw_app_link",
"lw_open_button",
"lw_sync_link",
"lw_about_link",
"lw_open_breach_link",
"mtr_report_link",
"mtr_about_link",
"mtr_signup_button"

Comment 10

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

(In reply to Erica Wright [:ewright] from comment #9)

:tdsmith :loines some of the names were too long, and un-allowed in the Yaml file, please note that I shortened them.

👍; thanks!

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6b05c82d94ca
Add basic telemetry to protection report. r=mtigley,johannh

Comment 12

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/6b05c82d94ca