1616356 - Record a telemetry event the first time password generation autocomplete is offered for a site

Status: VERIFIED FIXED

(Toolkit :: Password Manager, task, P1)

Description

[Matthew N. [:MattN]]

Currently we record telemetry when a generated password is filled but we don't know what the conversion rate is since we don't know how often it's offered. Knowing the conversion rate is useful to know if users are using the feature.

This is to fix https://sql.telemetry.mozilla.org/queries/65340#166315 so that we can calculate the percentage of times a user chooses to generate a password when given the opportunity.

Why only the first time for the site in a session? Because we don't want to count twice if there is a password confirmation field or the user gets an error and has to fill it again on the next page.

Comment 1

[Matthew N. [:MattN]]

It would be really helpful to uplift this since we will offer pwgen much more soon.

Comment 2

[Timea Cernea [:tbabos]]

Hey Matt,
This is the list of sites from Top 100 Alexa we use for every password feature testing. It would be nice to crosscheck this with the telemetry results just to be sure we are on the same line as the users regarding password generation usage. Let me know if this sounds good and is feasible with the telemetry results.

Comment 3

[Matthew N. [:MattN]]

(In reply to Timea Cernea [:tbabos] from comment #2)

This is the list of sites from Top 100 Alexa we use for every password feature testing. It would be nice to crosscheck this with the telemetry results just to be sure we are on the same line as the users regarding password generation usage. Let me know if this sounds good and is feasible with the telemetry results.

Do you mean that when you are testing those pages you could perhaps also check the telemetry from about:telemetry to ensure it matches what you see visually? If so, that sounds like a great way to verify this bug.

Comment 4

[Matthew N. [:MattN]]

This is where we show the autocomplete popup so we can check resultStyles.has("generatedPassword")

We can probably use a WeakSet to track if we recorded for a document already though this code is in the parent process so we can't use the content document.

Comment 5

[Bianca Danforth [:bdanforth]]

Oops sorry Sandy; no NI intended.

Comment 6

[Bianca Danforth [:bdanforth]]

Sorry I posted this on the wrong bug. Not having a good day today!

Comment 7

[Bianca Danforth [:bdanforth]]

Attachment: Bug 1616356 - Record a telemetry event the first time password generation autocomplete is shown for a site

WIP

Comment 8

[Bianca Danforth [:bdanforth]]

Attachment: bug-1616356-data-review-form.md

This is for the event telemetry to record when the password generation option is shown to the user in the autocomplete popup.

Comment 9

[Tim Smith 👨‍🔬 [:tdsmith]]

Comment on attachment 9132162 [details]
bug-1616356-data-review-form.md

I left a comment in Phabricator about the event description; once that's clarified, this will be good to go!

--

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, in Events.yaml and the probe dictionary.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the Firefox telemetry opt-out.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Bianca.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, interaction data.

5) Is the data collection request for default-on or default-off?

Default-on.

6) Does the instrumentation include the addition of **any *new* identifiers**?

No.

7) Is the data collection covered by the existing Firefox privacy notice? 

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

No, permanent collection.

9) Does the data collection use a third-party collection tool?

No.

Comment 10

[Matthew N. [:MattN]]

Bianca pointed out the disconnect between this bug and bug 1548878. I was proposing per-document here but bug 1548878 is per-site in a session so it seems better to align on the latter.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/386667c597a6
Record a telemetry event the first time password generation autocomplete is shown for a site r=MattN

Comment 12

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/386667c597a6

Comment 13

[Matthew N. [:MattN]]

Comment on attachment 9130908 [details]
Bug 1616356 - Record a telemetry event the first time password generation autocomplete is shown for a site

Beta/Release Uplift Approval Request

• User impact if declined: Users may be offered password generation in cases where it's inappropriate in Fx76 due to new heuristics (bug 1595244) but we won't have a baseline from Fx75 to compare to. This will also allow us to see how many more people are seeing password generation in Fx76.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: I have manually verified the data is coming in properly on the server and looks plausible but I didn't do much testing on the client:
  (Test this is part of our PI request.)
  Ensure the event pwmgr autocomplete_shown generatedpassword
  is only shown once per principal (site + container/PB flags) per session and isn't shown in other cases.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Just recording a telemetry event when password generation is shown. Unlikely to cause crashes or break the feature since the telemetry code runs last.
  Note: The one hunk that looks like a 27 line change is just deleting an if () that was always true so the indentation of the inside changed.
• String changes made/needed: None

Comment 14

[Julien Cristau [:jcristau]]

Comment on attachment 9130908 [details]
Bug 1616356 - Record a telemetry event the first time password generation autocomplete is shown for a site

telemetry for password generation, approved for 75.0b5

Comment 15

[Razvan Maries]

https://hg.mozilla.org/releases/mozilla-beta/rev/8c3aa7ef9e2c

Comment 16

[Bianca Danforth [:bdanforth]]

QA verification notes

• The new telemetry event in this bug should only be recorded once per site per session when the password generation autocomplete popup is shown.
• Note that the event will also be recorded when the password generation feature is used in the context menu, since that shows the autocomplete popup as well in order to display a warning label to users in certain situations.
• The event is recorded in the parent process.
• Each event should have a category of "pwmgr", a method of "autocomplete_shown" and an object of "generatedpassword". The value and extra fields should be blank.

Known issues

• If the autocomplete popup is shown without the password generation option, there may be an error logged to the Browser Console: TypeError: can't access property "autocompleteShown", generatedPW is undefined. This is being addressed in a follow-up patch as part of Bug 1619498 that we also hope to uplift to Beta 75.

Comment 17

[Timea Cernea [:tbabos]]

Verified-fixed on Firefox Nightly 76 (2020-03-18) and Beta 75.0b5 (64-bit) on Windows 10 x64, MacOS 10.13 and Ubuntu 16.04,
The telemetry event appears only once per session and per site in the parent process, regardless of how the Password Generation option is toggled (context menu, force or default "new-password" field).

Verified on both login and register forms for 10 sites, can be viewed in this doc

The only site on which the telemtry event is recorded twice is on Ebay, but that also happens with saved credentials on the login and register forms as they are saved for a different addresses (reg.ebay.com and signin-ebay.com)