1628849 - Collect experiment telemetry related to password import suggestion

Status: VERIFIED FIXED

(Toolkit :: Password Manager, enhancement, P1)

Description

[Ed Lee :Mardak]

Su has put together a telemetry requirements doc for password import autocomplete suggestion:
https://docs.google.com/document/d/1_q3RgyBY-2xJgdkyI3HAnooDtAMl-aM-Oq1vyFSFXJ8

With the new designs and triggering conditions from bug 1618311 comment 4 notably only showing the import suggestion when the user doesn't have a saved login for the site and within 7 days profile age…

I think a implementation for control and experiment branch is for when autocomplete is deciding what to show (e.g., user has focused a login field), we can report event telemetry with 2 numbers:

• how many saved logins firefox would show
• how many import suggestions could be shown (i.e., number of other browsers with saved login)

Where control branch never actually shows the import suggestion and import branch decides to show if no saved logins and profile age <1week.

Su, are those 2 values sufficient for analysis (in addition to import suggestion and import wizard interaction)? Is there a good way to send a pair of values for an event -- json/js array, e.g., [0,1] for 0 saved logins for this site and 1 browser has saved login; and after import or user saved login: [1,1]? I would think two separate event telemetry with saved=0 importable=1 might be harder to associate with each other ??

Comment 1

[Matthew N. [:MattN]]

You may be interested in https://searchfox.org/mozilla-central/rev/9120151ddb35f2d4c37bfe613a54a4f10a9a3dc5/toolkit/components/telemetry/Events.yaml#300,308 which currently reports what rows show in login autocomplete (when the first has 0 or 1 character).

Comment 2

[Ed Lee :Mardak]

[Tracking Requested - why for this release]: We'll need telemetry for the 77 experiment-only feature from bug 1618311.

Summarizing the telemetry and expected behavior:

• control branch should have the same form autocomplete behavior as feature turned off
• new experiment event telemetry to count login field impressions and how many saved and importable in both control and treatment similar to when PWMGR_FORM_AUTOFILL_RESULT is recorded but not reusing that histogram https://searchfox.org/mozilla-central/rev/7fd1c1c34923ece7ad8c822bee062dd0491d64dc/toolkit/components/passwordmgr/LoginManagerChild.jsm#2285-2287
• add importable as extra_keys to form_autocomplete/show/logins event telemetry https://searchfox.org/mozilla-central/rev/2bfe3415fb3a2fba9b1c694bc0b376365e086927/toolkit/components/telemetry/Events.yaml#279,288-297
• new experiment event telemetry counting importable items in popup for treatment (duplicate of the previous but is dedicated new telemetry)
• new experiment event telemetry recording actions on importable items (item click, info click, item enter, item delete)
• we'll want bug 1631130 to measure that users are using saved/imported logins
  -- ideally we also have telemetry that the user submitted a password (and not just using a saved login)

Comment 3

[Ed Lee :Mardak]

Attachment: Bug 1628849 - Collect experiment telemetry related to password import suggestion r?MattN!

Comment 4

[Ed Lee :Mardak]

Attachment: Data Review Request Form

Events.yaml changes are at https://phabricator.services.mozilla.com/D73390#change-F6M6PHCi3D7P

Attachment 9139571 [details] is a screenshot of the feature in the treatment branch.

Comment 5

[Ed Lee :Mardak]

Because extra_keys are limited to 10 (https://searchfox.org/mozilla-central/rev/e9131060a0322f6c181b4933ac74e6440b6e723d/toolkit/components/telemetry/build_scripts/mozparsers/parse_events.py#18) the patch in https://phabricator.services.mozilla.com/D73390 combines login and loginWithOrigin to a single value so that we can add importableLogin as the new 10th key.

Comment 6

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Comment on attachment 9144914 [details]
Data Review Request Form

Super sorry for the delay here. r+, thanks.

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, in Events.yaml.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the Firefox telemetry opt-out.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

n/a, temporary collection

4) Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, interaction data.

5) Is the data collection request for default-on or default-off?

Default-on.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

Ed Lee is responsible for renewing the collection before it expires if additional collection is needed.

9) Does the data collection use a third-party collection tool?

No.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ad0e2c2c3247
Collect experiment telemetry related to password import suggestion r=MattN,sfoster

Comment 8

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/ad0e2c2c3247

Comment 9

[Ed Lee :Mardak]

Robert, other than the telemetry added in this bug, there is a user facing change different from what Valentina verified in bug 1618311:
when signon.showAutoCompleteImport=control now it no longer shows the password autocomplete popup with just "View Saved Logins" (as noted in bug 1618311 comment 12) and should behave the same as when the pref is its default value (blank)

For the telemetry, it should record event telemetry viewable from about:telemetry#events-tab with "exp_import" category and have either "event" or "impression" as the method.

"impression" should trigger when the page loads with password fields as "formfill" object or when the password popup opens as "popup" object.

"event" should trigger when interacting with the import suggestions (which are only shown for signon.showAutoCompleteImport=import, e.g., "click" when clicking on the main item or "info" when clicking on the info icon.

Comment 10

[Robert Martin [:romartin], Ecosystem QA]

Hello @Ed,

I have verified that the telemetry pings are recorded and displayed correctly, according to comment 9, in about:telemetry#events-tab using the latest Firefox Nightly 78.0a1 (Build ID: 20200511094328) on Windows 10 x64 and macOS 10.15.

I have also verified that if the signon.showAutoCompleteImport pref value is set to control, the autocomplete popup is no longer displayed.

Comment 11

[Ed Lee :Mardak]

Comment on attachment 9144909 [details]
Bug 1628849 - Collect experiment telemetry related to password import suggestion r?MattN!

Beta/Release Uplift Approval Request

• User impact if declined: Unchanged for most users who get default off. For those in the experiment both control and treatment, new telemetry is needed for successful analysis
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: comment 9 as verified in comment 10
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Adding telemetry that's only enabled for those in the experiment
• String changes made/needed: none

Comment 12

[Pascal Chevrel:pascalc]

Comment on attachment 9144909 [details]
Bug 1628849 - Collect experiment telemetry related to password import suggestion r?MattN!

Looks low risk in early betas and QA verified on nightly, uplift approved for beta, thanks.

Comment 13

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/74305c074d93

Comment 14

[Robert Martin [:romartin], Ecosystem QA]

I have verified that the telemetry pings are recorded and displayed correctly, according to comment 9, in about:telemetry#events-tab using the latest Firefox Beta 77.0b5 (Build ID: 20200512163137) on Windows 10 x64 and macOS 10.15.

I have also verified that if the signon.showAutoCompleteImport pref value is set to control, the autocomplete popup is no longer displayed.