1545172 - Enable recording of metrics into pre-account ecosystem ping/store

Status: RESOLVED FIXED

(Toolkit :: Telemetry, enhancement, P2)

Description

[Leif Oines [:loines]]

As requested in #1529233, I will update this bug with a patch and a list of metrics soon, then add the data review.

Comment 1

[Leif Oines [:loines]]

Attachment: Bug 1545172 - Enable recording of metrics into pre-account ecosystem ping/store r?janerik

Comment 2

[Leif Oines [:loines]]

List of metrics that I am enabling for the pre-account ping in this patch (jan erik already added total_uri_count):

scalars:

• active_ticks

histograms:

• SEARCH_COUNTS
• PWMGR_BLOCKLIST_NUM_SITES
• PWMGR_LOGIN_LAST_USED_DAYS
• PWMGR_LOGIN_PAGE_SAFETY
• PWMGR_MANAGE_COPIED_PASSWORD
• PWMGR_MANAGE_COPIED_USERNAME
• PWMGR_MANAGE_DELETED
• PWMGR_MANAGE_OPENED
• PWMGR_MANAGE_VISIBILITY_TOGGLED
• PWMGR_NUM_PASSWORDS_PER_HOSTNAME
• PWMGR_NUM_SAVED_PASSWORDS
• PWMGR_PROMPT_REMEMBER_ACTION
• PWMGR_PROMPT_UPDATE_ACTION
• PWMGR_SAVING_ENABLED

This covers the scalars and histograms that were approved by policy in this proposal to go into the pre-account ping.

Comment 3

[Leif Oines [:loines]]

Attachment: Data collection review for data steward

Tim feel free to pass it to someone else

Comment 4

[Leif Oines [:loines]]

Comment on attachment 9059108 [details]
Data collection review for data steward

Here's the text of the review request:

Request for data collection review form
All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.

Bugzilla 
What questions will you answer with this data?
See this doc for much more detail, but the short answer is that this is half of the piece of ecosystem telemetry on desktop that will allow us to answer how signing up to an FxA account affects browsing behavior. However this patch alone will not allow us to answer these questions - lots of other work will need to be done for this data to be useable.

Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:
Mozilla needs to answer these questions because it will help us understand the impact of Account usage on browsing behavior, and eventually it will allow us to combine measurements from mobile and desktop which we can’t do currently.  

What alternative methods did you consider to answer these questions? Why were they not sufficient?
We currently have no way of measuring the true impact of account usage on browsing behavior or across-app usage at all so there are no other alternatives.

Can current instrumentation answer these questions?
No, this is a new type of cross-platform telemetry and all existing telemetry is platform specific. 

List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories on the Mozilla wiki.
NOTE NONE OF THE MEASUREMENTS LISTED BELOW ARE NEW, THEY ARE PRE-EXISTING AND ARE JUST BEING PACKAGED INTO A NEW PING. The new ping will eventually have two new identifiers (ecosystem_client_id and FxA uid) but those have not been implemented yet (will request another data review for that when the time comes)
    The list measurements that we are adding the pre-account ping are below. They are the ones that have prior approval by policy for inclusion in ecosystem telemetry. 

scalars:
active_ticks

histograms:
SEARCH_COUNTS
PWMGR_BLOCKLIST_NUM_SITES
PWMGR_LOGIN_LAST_USED_DAYS
PWMGR_LOGIN_PAGE_SAFETY
PWMGR_MANAGE_COPIED_PASSWORD
PWMGR_MANAGE_COPIED_USERNAME
PWMGR_MANAGE_DELETED
PWMGR_MANAGE_OPENED
PWMGR_MANAGE_VISIBILITY_TOGGLED
PWMGR_NUM_PASSWORDS_PER_HOSTNAME
PWMGR_NUM_SAVED_PASSWORDS
PWMGR_PROMPT_REMEMBER_ACTION
PWMGR_PROMPT_UPDATE_ACTION
PWMGR_SAVING_ENABLED


Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.

How long will this data be collected? Choose one of the following:
I want to permanently monitor this data. 
Contact: [e-mail]

What populations will you measure?
Which release channels? All eventually but landing in nightly now
Which countries? All
Which locales? All
No other filters

If this data collection is default on, what is the opt-out mechanism for users?
User can opt out of this telemetry by turning telemetry off generally or by a pref toolkit.telemetry.ecosystemtelemetry.enabled

Please provide a general description of how you will analyze this data.
When the rest of ecosystem telemetry lands we will use it to aggregate telemetry across clients and to measure changes in usage before and after account creation. 

Where do you intend to share the results of your analysis?
Only with Mozilla employees/NDA mozillians

Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? 
Not currently, but ecosystem telemetry may eventually be sent to amplitude in the medium-to-long term future.

Comment 5

[Leif Oines [:loines]]

One other thing I neglected to add in the data review request: the pwmgr probes are included for the purposes of Lockwise (formerly Lockbox) so that password manager usage can be measured across devices/apps.

Comment 7

[Alicia Gray]

For data steward review: Confirming SEARCH_COUNTS probe is category 2 for pre-account ecoysystem telemetry pings just like in the main ping.

Comment 8

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Comment on attachment 9059108 [details]
Data collection review for data steward

data-review+

-- 

Some context: the pre-account ping was implemented along with its first probe (`total_uri_count`) in bug 1529232. The ping itself is documented at https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/data/ecosystem-telemetry.html. Per bug 1530655, it still lacks an ecosystem client ID.

1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, in Histograms.json and Scalars.yaml.

2) Is there a control mechanism that allows the user to turn the data collection on and off? Provide details as to the control mechanism available.

Yes, the Firefox telemetry opt-out.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Leif Oines.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

All listed probes are Category 2, interaction data.

5) Is the data collection request for default-on or default-off?

Default-on.

6) Does the instrumentation include the addition of **any *new* identifiers**?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No)

No.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0f774c6d8ac3
Enable recording of metrics into pre-account ecosystem ping/store r=janerik

Comment 10

[Leif Oines [:loines]]

i think we're done here

Comment 11

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/0f774c6d8ac3