1597895 - Data Collection Review Request: application-services Logins Store quality metrics

Status: RESOLVED FIXED

(Firefox :: Sync, defect)

Description

[Ryan Kelly [:rfkelly]]

The application-services Logins Store is a re-usable component for storing and syncing saved logins, currently used in the Lockwise mobile apps and coming soon to Firefox Preview:

https://github.com/mozilla/application-services/tree/master/components/logins

Since this is a new component underlying an important piece of browser functionality, we'd like to gather some basic quality and performance metrics so we can be confident it is working correctly. A skeleton of the metrics-gathering code (including the proposed Glean metrics.yaml file) is being developed here:

https://github.com/mozilla/application-services/pull/2225

This bug is to track review of the proposed telemetry collection, and I'll attach the review request template presently...

Comment 1

[Ryan Kelly [:rfkelly]]

Attachment: data-review-request.md

Comment 2

[Ryan Kelly [:rfkelly]]

:Dexter, if you have a moment, could you please take a look at the attached review request and the related metrics.yaml in https://github.com/mozilla/application-services/pull/2225, in case we seem to be on the wrong track from a technical perspective? (Some of the metrics in that metrics.yaml are not going to work as-is, for reasons explained therein; but hopefully it will demonstrate what we want to achieve).

Comment 3

[Alessio Placitelli [:Dexter]]

Thanks for flagging me! The metrics look reasonable, I left a bunch of comments in the PR. For the data-review, it LGTM, but please flag a data-steward once the PR is settled :)

Comment 4

[Ryan Kelly [:rfkelly]]

Comment on attachment 9110161 [details]
data-review-request.md

Canceling data review request for now, since I don't think we understand enough about how the proposed "num_saved_passwords" metric will behave. I'll post a revised version that does not contain this metric.

Comment 5

[Ryan Kelly [:rfkelly]]

Attachment: data-review-request.md

Here's a revised request with a smaller number of more clearly-scoped metrics.

Comment 6

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Comment on attachment 9110451 [details]
data-review-request.md

Picking this up in response to the fx-data-stewards post; thanks for reaching out.

The intended points of articulation for Glean collections in reusable components versus the data stewardship program are not clear to me, but we don't have to hold this request hostage; these look uncontroversial under any likely standard. I opened https://bugzilla.mozilla.org/show_bug.cgi?id=1600399 to discuss.

AIUI—someone may correct me—the scope of this review is to instrument these metrics in a library using Glean. Additional reviews are necessary to integrate this code into any user-facing product.

--

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, in components/logins/android/metrics.yaml.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Embedding applications must provide a control to disable collection.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

n/a

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

These round up to Category 2, interaction data.

5) Is the data collection request for default-on or default-off?

Default-on.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

:rfkelly is responsible for renewing the collection before it expires.

9) Does the data collection use a third-party collection tool?

No.

Comment 7

[Ryan Kelly [:rfkelly]]

Thanks :tdsmith!

AIUI—someone may correct me—the scope of this review is to instrument these metrics in a library using Glean.
Additional reviews are necessary to integrate this code into any user-facing product.

In Bug 1598247 Comment 1, :Dexter suggests that we consider requesting a "blanket review" to cover any future potential consuming applications. Given that:

these look uncontroversial under any likely standard

Is a "blanket review" a sufficiently-well-defined thing that I could ask for, either here or in a fresh bug?

2. Is there a control mechanism that allows the user to turn the data collection on and off?
   Embedding applications must provide a control to disable collection.

I'll add for completeness: these are deliberately going in Glean's default metrics ping, so embedding applications won't need to provide any explicit toggle specifically for these metrics, but rather they must provide one Glean metrics as a whole.

Comment 8

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

(In reply to Ryan Kelly [:rfkelly] from comment #7)

In Bug 1598247 Comment 1, :Dexter suggests that we consider requesting a "blanket review" to cover any future potential consuming applications. Given that:

these look uncontroversial under any likely standard

Is a "blanket review" a sufficiently-well-defined thing that I could ask for, either here or in a fresh bug?

We haven't described a process for that and I'm reluctant to freelance one until I understand what we expect from a per-application collection review; sorry! I believe I understand how to review for Firefox Preview. I'm not sure about Lockwise.

I notice that the data review documentation and forms say "Firefox" in a couple of places. https://support.mozilla.org/en-US/kb/firefox-lockwise-and-privacy seems to suggest that, while Lockwise requires using Firefox, that Lockwise has its own privacy promises.

Alicia, is Lockwise covered by the Firefox Privacy Notice? Can we review collections for Lockwise the same way as we do for collections for the desktop browser?

Comment 9

[Alessio Placitelli [:Dexter]]

Side note: if these metrics are only going in Lockwise, then it might make sense to implement them directly in that product.

Comment 10

[Ryan Kelly [:rfkelly]]

if these metrics are only going in Lockwise, then it might make sense to implement them directly in that product.

It's certainly my intention for these metrics to be in multiple products, most notably Fenix and Lockwise.

Comment 11

[Alicia Gray]

(In reply to Tim Smith 👨‍🔬 [:tdsmith] from comment #8)

(In reply to Ryan Kelly [:rfkelly] from comment #7)

In Bug 1598247 Comment 1, :Dexter suggests that we consider requesting a "blanket review" to cover any future potential consuming applications. Given that:

these look uncontroversial under any likely standard

Is a "blanket review" a sufficiently-well-defined thing that I could ask for, either here or in a fresh bug?

We haven't described a process for that and I'm reluctant to freelance one until I understand what we expect from a per-application collection review; sorry! I believe I understand how to review for Firefox Preview. I'm not sure about Lockwise.

I notice that the data review documentation and forms say "Firefox" in a couple of places. https://support.mozilla.org/en-US/kb/firefox-lockwise-and-privacy seems to suggest that, while Lockwise requires using Firefox, that Lockwise has its own privacy promises.

Alicia, is Lockwise covered by the Firefox Privacy Notice? Can we review collections for Lockwise the same way as we do for collections for the desktop browser?

HI Tim,
Lockwise is covered under the Firefox Privacy Notice and the data collection has to adhere to all of our standard policies and practices. We should be able to review collections for Lockwise the same way we do for the desktop browser. BUT, that being said, if we run across something that is unique and doesn't seem to fit into one of our current defined categories, feel free to escalate to Trust and we can help assess.

The steward steering committee and I are trying to develop a way to incorporate all of our products/services into this review process so we are revisiting the data categories to see how they should be modified to incorporate some of these other things we are seeing come into play. Does this help?

Comment 12

[Ryan Kelly [:rfkelly]]

I'm going to re-open this based on https://bugzilla.mozilla.org/show_bug.cgi?id=1600399#c6 and https://github.com/mozilla/application-services/pull/2325#discussion_r354859025; it seems clear from those discussions that we can't land these metrics in the component without data-review+ for all applications that currently use that component. In our case, those applications are:

• Firefox Preview
• Firefox Lockwise for Android

Tim, do you feel like ensuing discussions have clarified how to proceed for those applications? What's the right way to move forward on them, should I make up a fresh review-request template? (Alternately, let me know if you'd prefer to delegate this to :chutten based on the conversation from Bug 1600399 😄)

Comment 13

[Alessio Placitelli [:Dexter]]

(In reply to Ryan Kelly [:rfkelly] from comment #12)

I'm going to re-open this based on https://bugzilla.mozilla.org/show_bug.cgi?id=1600399#c6 and https://github.com/mozilla/application-services/pull/2325#discussion_r354859025; it seems clear from those discussions that we can't land these metrics in the component without data-review+ for all applications that currently use that component. In our case, those applications are:

• Firefox Preview
• Firefox Lockwise for Android

Sorry for the unsolicited chiming in :) As I understand Chris' comment, that's not the case. The data-review in this component is for collecting the metrics with the component, with a potential target population of all the products using Glean and using the component.

When using this component in either Firefox Preview or Lockwise, another data-review is required, for using the collection from the component.

Since I'm not chutten, I might be misinterpreting this as well. So, asking Chris :)

Comment 14

[Chris H-C :chutten]

Dexter has it. When the collection is added to a component, the component should file a Data Review with the knowledge you know at the time. When a component with data is added to an application, the application should file a Data Review for the expanded population that the new application hopes to reach.

That's it.

This is functionally the same thing we do with a single data review on Desktop. If we suddenly started shipping a new product with Firefox embedded in it, then that new product should file a Data Review explaining the expanded population it hopes to reach because the embedded Firefox (acting as a component in this hypothetical) will be collecting data.

Does that make sense, Ryan?

Comment 15

[Ryan Kelly [:rfkelly]]

When using this component in either Firefox Preview or Lockwise, another data-review is required, for using the collection from the component.

The part that's tripping me up here is that the "when" in the above is "right now" - they're both already using the logins-store component, just an earlier version that doesn't yet include the metrics, and they'll pick up the new version with the metrics in a fairly automated fashion.

So in general:

When the collection is added to a component, the component should file a Data Review with the knowledge you know at the time.

This works for me, and I appreciate the simplicity of this approach! I think I have a much improved understanding of how this process should work in future, thanks for helping me navigate it.

However, in this specific case, my read of Comment 6 was that Tim's data-review+ explicitly did not factor in the embedding applications, because it wasn't clear our policy/process for doing so. Thus I wasn't confident that this bug met our "the component should file a Data Review with the knowledge you know at the time" bar, yet.

So, to ask that final clarifying question directly: does this bug suffice for data-review of the logins metrics in existing consumers of the logins component (Lockwise and Firefox Preview), or should I coordinate a separate one?

Comment 16

[Ryan Kelly [:rfkelly]]

(Given the "Data Review is forgiving" sentiment from this comment, I'm going to proceed under the assumption that this bug does indeed suffice, or at worst can be made to suffice without too much difficulty, since I'm getting a very strong (and welcome!) vibe of "you don't need to be blocked on this" from the surrounding discussion).

Comment 17

[Chris H-C :chutten]

I'll throw to :tdsmith for the last okay, but that is my understanding as well. This looks good and correct to me, and at the very least combined with all the words we've collectively written on the subject no one can doubt we're at least trying to get this right : )

Comment 18

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Yes. The target population is all Lockwise and Firefox Preview users. I think a revised response looks like:

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, in components/logins/android/metrics.yaml in the mozilla/application-services repository. Hopefully there will also be a Glean probe dictionary soon!

2. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. Firefox Preview has a usage and technical data control. Lockwise also provides a toggle in Settings.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

n/a

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

These round up to Category 2, interaction data.

5. Is the data collection request for default-on or default-off?

Default-on.

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

7. Is the data collection covered by the existing Firefox privacy notice?

Yes.

8. Does there need to be a check-in in the future to determine whether to renew the data?

:rfkelly is responsible for renewing the collection before it expires.

9. Does the data collection use a third-party collection tool?

No.