1548880 - Add telemetry for when a field with a generated password is edited by the user

Status: VERIFIED FIXED

(Toolkit :: Password Manager, task, P2)

Description

[Matthew N. [:MattN]]

We want to how often users need to make corrections to our generated passwords.

Comment 1

[Matthew N. [:MattN]]

For the initial version we can just record the first edit to a field after generation and store that boolean state in the generated password state of LoginManagerParent.

We should use event telemetry just like https://searchfox.org/mozilla-central/rev/22b330ecb3edba1536a54887060cbdd09db21c59/toolkit/components/telemetry/Events.yaml#438-447 and add tests too.

This can build on top of https://phabricator.services.mozilla.com/D38220

Comment 2

[Matthew N. [:MattN]]

Data review will be required like we did on bug 1548878.

Comment 3

[Tim Nguyen :ntim]

Attachment: Bug 1548880 - Add telemetry for when a field with a generated password is edited by the user.

Comment 4

[Tim Nguyen :ntim]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=fb0c4063590982e2ab7df0b519fda9bb8947b7cd

Comment 5

[Tim Nguyen :ntim]

Attachment: data-review-request-form.md

Comment 6

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Data steward hat: I don't think I understand the circumstances under which an event is sent; the event documentation should be clearer. Is this sent when a generated password is edited before it's submitted, on the page where it was generated? Is it sent if a user edits the filled password while attempting to log in at a later date? Is it sent if a user edits a saved password using the password manager UI?

Data scientist hat: this is outside the scope of data-review, but what (existing or planned) probe should we compare this to in order to understand how often this happens?

Thanks!

Comment 7

[Matthew N. [:MattN]]

(In reply to Tim Smith 👨‍🔬 [:tdsmith] from comment #6)

Data steward hat: I don't think I understand the circumstances under which an event is sent; the event documentation should be clearer. Is this sent when a generated password is edited before it's submitted, on the page where it was generated?

That would be the most common case but it's not limited to the same page as the user could end up on an error page or another page where they need it again and may choose to use the generated password row instead of the saved login row (it would have been automatically saved in most cases).

Is it sent if a user edits the filled password while attempting to log in at a later date?

The generated password is only around for the life of the Firefox session so only if they filled the "generated password" instead of the automatically saved login from a login form in the same Fx session. Once it's filled it also becomes a Saved Login (in most cases) and we aren't currently measuring edits to filled saved logins/passwords.

Is it sent if a user edits a saved password using the password manager UI?

No, hence the occurence of "field" in the event method.

I'll work with ntim to improve the description.

Data scientist hat: this is outside the scope of data-review, but what (existing or planned) probe should we compare this to in order to understand how often this happens?

We would compare it to pwmgr/autocomplete_field/generatedpassword as that tells us the first time a user uses a specific generated password. Dividing pwmgr/filled_field_edited/generatedpassword by pwmgr/autocomplete_field/generatedpassword would tell us the percent of generated passwords that the user edited in the field and therefore it likely didn't meet their needs.

Comment 8

[Matthew N. [:MattN]]

Attachment: data-review-request-form v2

We are updating the Events.yaml description to "The first time each generated password filled in a website form field is edited by the user in a field it was filled in"

Comment 9

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Comment on attachment 9080125 [details]
data-review-request-form v2

Thanks; I think the comments in this bug about what a "generated password" is (as distinct from a saved password that was once generated) and its lifetime will help supplement the event documentation.

This is approved as-is for a permanent collection. It does look like the question you're posing is related to whether the current design of the feature is fit for purpose. That doesn't sound like something that we need to continue tracking indefinitely, so you could consider adding an expiry date.

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, in Events.yaml.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the Firefox telemetry opt-out.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Leif.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, interaction data.

5) Is the data collection request for default-on or default-off?

Default-on.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No)

No.

9) Does the data collection use a third-party collection tool? **If yes, escalate to legal.**

No.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d8bed1c7c0d3
Add telemetry for when a field with a generated password is edited by the user. r=MattN

Comment 11

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/d8bed1c7c0d3

Comment 12

[Daniel Bodea [:danibodea]]

How can I manually verify the implementation of the telemetry when the user edits a generated password?
Thanks!

Comment 13

[Matthew N. [:MattN]]

You can look at about:telemetry#events-tab and see that an event gets recorded for the first edit for a given generated password. Multiple edits to the same password should only get 1 event.

Comment 14

[Daniel Bodea [:danibodea]]

I have verified this implementation on Windows 10, Mac OS 10.14.5 and Ubuntu 18.04 on Nightly v70.0a1 from 2019-08-06.
It appears that password generations and generated password edits are being tracked correctly with instant telemetry events, but only ONCE for ANY SITE in EVERY ONE SESSION. Is this intended? If yes, then this implementation bug can be marked as verified. Thanks!

Comment 15

[Daniel Bodea [:danibodea]]

Attachment: password generated and password edit actions telemetry events.png

Are the events correctly displayed? Is there no "Value" or "Extra" information needed?

Comment 16

[Matthew N. [:MattN]]

(In reply to Bodea Daniel [:danibodea] from comment #14)

I have verified this implementation on Windows 10, Mac OS 10.14.5 and Ubuntu 18.04 on Nightly v70.0a1 from 2019-08-06.
It appears that password generations and generated password edits are being tracked correctly with instant telemetry events, but only ONCE for ANY SITE in EVERY ONE SESSION. Is this intended? If yes, then this implementation bug can be marked as verified. Thanks!

Yes, that is correct, we only record the first fill or edit for a site within a session.

(In reply to Bodea Daniel [:danibodea] from comment #15)

Are the events correctly displayed? Is there no "Value" or "Extra" information needed?

Yes, we don't have any other data to record.