1650929 - [Doorhanger V2] Add telemetry event for username/password edits in the doorhanger

Status: VERIFIED FIXED

(Toolkit :: Password Manager, task, P2)

Description

[:severin Rudie]

Add telemetry that reports how frequently PM doorhanger information is edited, to support our OKR of improving our recommended values.

Requirements

• When the user selects 'save' or 'update', send an event with booleans 'did_edit_un' and 'did_edit_pw'
• Typing or pasting into a field should set the did_edit_* boolean state to true
• State should be reset for each doorhanger instance
• No ID (or other information that could be correlated to a user) should be sent

Verification notes

• Can be verified locally in about:telemetry (There are some edge cases that affect when about:telemetry gets updated. You probably know more about this than I do, but if not feel free to reach out.)
• The event will be called doorhanger_submitted. See its definition for details
• Ensure that the extra_keys did_edit_un and did_edit_pw match user behavior
• Ensure that the sent object (save or update) matches the original state when the doorhanger was created (this will not change if the user modifies the doorhanger fields)
• Ensure that exactly 1 event is sent every time the doorhanger is either saved or updated

Comment 1

[:severin Rudie]

Attachment: Bug 1650929 - add 'pwmgr doorhanger modified' telemetry; r=MattN

Comment 2

[:severin Rudie]

Attachment: Data review request

Comment 3

[Tim Smith 👨‍🔬 [:tdsmith]]

Comment on attachment 9163647 [details]
Data review request

lgtm; thanks!

--

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, in Events.yaml and the probe dictionary.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the Firefox telemetry opt-out.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

n/a; temporary collection

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, interaction data

5) Is the data collection request for default-on or default-off?

Default-on.

6) Does the instrumentation include the addition of **any *new* identifiers**?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

It's up to :severin to renew the collection as needed.

9) Does the data collection use a third-party collection tool?

No.

Comment 4

[:severin Rudie]

Comment on attachment 9161785 [details]
Bug 1650929 - add 'pwmgr doorhanger modified' telemetry; r=MattN

Beta/Release Uplift Approval Request

• User impact if declined: No user impact (MattN addition: we won't be able to measure the impact of our doorhanger v2 improvements)
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is a telemetry only change. It is very low risk, and should be transparent to the user.

We have no alternative to uplifting this if we want to establish baseline numbers for one of our OKRs.

• String changes made/needed: None

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/64eaf3f34725
add 'pwmgr doorhanger modified' telemetry; r=MattN

Comment 6

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/64eaf3f34725

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Have we verified that the incoming Telemetry data from Nightly looks sane so far?

Comment 8

[Matthew N. [:MattN]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #7)

Have we verified that the incoming Telemetry data from Nightly looks sane so far?

Yes: https://sql.telemetry.mozilla.org/queries/72863

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9161785 [details]
Bug 1650929 - add 'pwmgr doorhanger modified' telemetry; r=MattN

Thanks for the confirmation. Approved for 79.0 RC1.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/cc22eb08be8d

Comment 11

[:severin Rudie]

I updated the comment 0 with verification details. LMK if any of it could use clarification.

Comment 12

[Timea Cernea [:tbabos]]

Hey Severin,
I am getting "did_edit_un:true" for Save and Update doorhanger for when I select an option from the username autocomplete dropdown, the following 2 scenarios are not ok at this moment:

Open doorhanger -> select autocomplete value -> save == did_edit_un: false | Recording of the issue: https://streamable.com/hhblod
Open doorhanger -> modify username -> select autocomplete value -> save == did_edit_un: false | Recording of the issue: https://streamable.com/n8da95

Same goes for the Update scenarios.

Comment 13

[:severin Rudie]

That's my bad Timea, I conflated two bugs when I described those cases to you. These two cases won't need to be valid until bug 1650941 lands, which will update the telemetry to account for the new autocomplete functionality.

Comment 14

[Timea Cernea [:tbabos]]

No worries, I also missed to see Bug 1650941 to figure it out on my own.
Verified-fixed on latest Firefox Nightly 80 and Beta 79 on Windows 10, MacOS and Ubuntu 16.04 for the cases where we manually edit the username or password.