1654388 - Implement Prefs (if not existed yet) for A/B test survey analysis

Status: VERIFIED FIXED

(Toolkit :: Form Autofill, task, P3)

Description

[chsiang]

As we run cc autofill A/B test, we will also survey users to understand the reason behind their actions. We will be able to get the most insights if we have proper prefs implemented for the survey analysis.

Below is a list of requirements from the UR team. Please investigate and see if these prefs (not telemetry) already exist. If not, let's prioritize them for Release 80.

To segment users based on their actions:

• User chooses "Save Credit Card"
• User chooses "Don't Save"
• User chooses Never Seen Again
• User dismisses the doorhanger by clicking outside, close the tab, or navigate away

It's best if we can keep a count on each actions. But if it's too complicated, let's have a pref for the "latest" actions.

To further analyze how a user's trust might impact their decisions:

• User has at least one address saved

• User has at least one password (login) saved

• User unchecked "Ask to save logins and passwords for websites"

• User unchecked "Autofill addresses

• User autofilled an address in the past month

• User autofilled logins and passwords in the last month

I'm not sure if the last two are possible. If it's too much hassle, we are ok using other signals.

Comment 1

[Jim Mathies [:jimm]]

We're going to investigate the best way to get this information for the survey.

We won't have this in 80, but we should be able to get it into 81 so tracking in the mvp list.

Comment 2

[Adam Roach [:abr]]

(In reply to chsiang from comment #0)

As we run cc autofill A/B test, we will also survey users to understand the reason behind their actions. We will be able to get the most insights if we have proper prefs implemented for the survey analysis.

Below is a list of requirements from the UR team. Please investigate and see if these prefs (not telemetry) already exist.

I'm trying to figure out what context we might need these values in. My understanding is that we'll be using Normandy to target users for the survey, and I'm reading this bug as asking to make sure we have enough information to perform that targeting.

This makes a pretty big difference in what needs to be implemented. If these are only used for Normandy rules, then the telemetry that is being uplifted to 80 provides much of this data already, and Normandy can use telemetry as part of its targeting rules (see https://mozilla.github.io/normandy/user/filters.html#normandy.telemetry).

On the other end of the spectrum: if we're not using these for Normandy, but just need to record and access them, then it might make sense to record them in a way that doesn't add more keys to the preferences store unnecessarily (or, minimally, stores them all in a single pref in a format that could be easily accessed, but which wouldn't necessarily work for Normandy rules).

Comment 3

[chsiang]

I believe we plan to use the prefs for analysis. The survey will be instrumented through SurveyGIzmo and I'm not sure how SurveyGizmo and Normandy intersect. I'll let Kamyar comment on the details.

Comment 4

[Kamyar Ardekani (he/him)]

Yes, this is for targeting and segmenting surveys. Telemetry CAN work too! The limitation of Telemetry is that it can only target based on the ping corresponding to the user's current session and not past behavior. For example, if a user has dismissed the doorhanger in their most recent session, but in the past they have saved cc information, Normandy will target based on the latest ping which says the doorhanger was dismissed. That being said, I'm not completely familiar with how the pings for this are implemented. Do you have a link to what pings are involved and what they capture?

Comment 5

[Adam Roach [:abr]]

Attachment: Bug 1654388: Part 1: Track doorhanger interaction r=zbraniecki!

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/28d98cbf8160
Part 1: Track doorhanger interaction r=zbraniecki

Comment 7

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/28d98cbf8160

Comment 8

[Adam Roach [:abr]]

Attachment: Bug 1654388: Part 2: Record address and password usage r=zbraniecki!

Comment 9

[Adam Roach [:abr]]

With these two patches, the requested information is exposed via the following prefs:

Pref	Action
extensions.formautofill.creditCards.usage.saveCc	User chooses "Save Credit Card"
extensions.formautofill.creditCards.usage.cancelCcSave	User chooses "Don't Save"
extensions.formautofill.creditCards.enabled*	To calculate: User chooses Never Seen Again
extensions.formautofill.creditCards.usage.dismissed	User dismisses the doorhanger by clicking outside, close the tab, or navigate away
extensions.formautofill.addresses.usage.hasEntry	User has at least one address saved
signon.usage.hasEntry	User has at least one password (login) saved
signon.rememberSignons*	To calculate: User unchecked "Ask to save logins and passwords for websites"
extensions.formautofill.addresses.enabled*	To calculate: User unchecked "Autofill addresses
extensions.formautofill.addresses.usage.lastUsed	Last time address was autofilled (to calculate: User autofilled an address in the past month)
signon.usage.lastUsed	Last time password was filled (to calulate: User autofilled logins and passwords in the last month)

Those prefs marked "*" already existed prior to this bug.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/558dce220fdd
Part 2: Record address and password usage r=zbraniecki

Comment 11

[Razvan Maries]

Backed out for perma failures.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=MJKlXErcTiO04rLnW5civQ.0&resultStatus=testfailed%2Cbusted%2Cexception&revision=61871a38eb0ff8d9ebdc28db10f03e74e01912a3&searchStr=windows%2C10%2Cx64%2Copt%2Cmochitests%2Ctest-windows10-64%2Fopt-mochitest-plain-e10s%2C4&failure_classification_id=2

Log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=312688586&repo=autoland&lineNumber=8350

Backout: https://hg.mozilla.org/integration/autoland/rev/3fa98e681edd27957e97152d541e0be28122d781

Comment 12

[Adam Roach [:abr]]

This new patch should fix the testing failure. I'm not landing it because there's some ongoing conversation about whether we want to move forward with the A/B testing at this time. I'm also reassigning the bug to Zibi so he can make sure it lands if/when it becomes appropriate to do so.

Comment 13

[Zibi Braniecki [:zbraniecki][:gandalf]]

setting NI on Jim to verify that we are moving forward with A/B testing.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7ce87f1b78b8
Part 2: Record address and password usage r=zbraniecki

Comment 15

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/7ce87f1b78b8

Comment 16

[Zibi Braniecki [:zbraniecki][:gandalf]]

Comment on attachment 9168347 [details]
Bug 1654388: Part 2: Record address and password usage r=zbraniecki!

Beta/Release Uplift Approval Request

• User impact if declined: We will not be able to run the A/B test study for Shirley as planned.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Please, verify https://bugzilla.mozilla.org/show_bug.cgi?id=1654388#c9
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This change is only adding temporary prefs for Fx 80 study
• String changes made/needed: None

Comment 17

[Julien Cristau [:jcristau]]

Comment on attachment 9168347 [details]
Bug 1654388: Part 2: Record address and password usage r=zbraniecki!

We already built the 80 release candidate, this is likely too late.

Comment 18

[Jim Mathies [:jimm]]

(In reply to Julien Cristau [:jcristau] from comment #17)

Comment on attachment 9168347 [details]
Bug 1654388: Part 2: Record address and password usage r=zbraniecki!

We already built the 80 release candidate, this is likely too late.

I've reached out to Julien to find out what the implications are here.

Comment 19

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9167416 [details]
Bug 1654388: Part 1: Track doorhanger interaction r=zbraniecki!

Not needed for 80 per jimm.

Comment 20

[Cosmin Muntean [:cmuntean], Ecosystem QA]

I have verified this issue on the latest Nightly 81.0a1 en-US build (Build ID: 20200821033746) on Windows 10 x64, macOS 10.15.6 and Ubuntu 20.04 x64. In order to verify this issue I have used the following scenarios:

1. Verified all the prefs from comment 9 on a new clean profile and check if they have the correct value. Here are the states of each pref:
   extensions.formautofill.creditCards.usage.saveCc - not created yet on new profiles
   extensions.formautofill.creditCards.usage.cancelCcSave - not created yet on new profiles
   extensions.formautofill.creditCards.enabled = true
   extensions.formautofill.creditCards.usage.dismissed - not created yet on new profiles
   extensions.formautofill.addresses.usage.hasEntry - false
   signon.usage.hasEntry - false
   signon.rememberSignons - true
   extensions.formautofill.addresses.enabled - true
   extensions.formautofill.addresses.usage.lastUsed - not created yet on new profiles
   signon.usage.lastUsed - not created yet on new profiles

2. I have saved a credit card and verified the pref:
   extensions.formautofill.creditCards.usage.saveCc = 1; Saved another credit card and the pref's value has increased;

3. I have submitted a credit card form, clicked the "Don't Save Credit Card" from doorhanger and verified the pref:
   extensions.formautofill.creditCards.usage.cancelCcSave = 1; Repeated the step and the pref's value has increased;

4. I have submitted a credit card form, clicked the "Never save Credit Cards" button from the doorhanger and verified the pref:
   extensions.formautofill.creditCards.enabled = false

• the doorhanger is no longer displayed
• the doorhager is redisplayed if the pref is switched back to "true".

5. I have submitted a credit card form and dismissed by clicking outside, close the tab, or navigate away:
   extensions.formautofill.creditCards.usage.dismissed = 1; The value correctly increased every time the doorhanger was dismissed.

6. I have completed a payment form, saved the address, and verified the pref:
   extensions.formautofill.addresses.usage.hasEntry = true

7. I have saved a password on a website and verified the pref:
   signon.usage.hasEntry = true

8. I have unchecked the "Ask to save logins and passwords for websites" checkbox from about:preferences#privacy and verified the pref:
   signon.rememberSignons = false

9. I have unchecked the "Autofill addresses" checkbox from about:preferences#privacy and verified the pref:
   extensions.formautofill.addresses.enabled = false

10. I have used the saved address on a payment form and verified the pref:
    extensions.formautofill.addresses.usage.lastUsed = 1598010440 (timestamp current date)

11. I have logged in on a website with a saved login and checked the pref:
    signon.usage.lastUsed = 1598002009 (timestamp current date)

I haven't encountered any issues while testing. Please let me know if there are any other scenarios that I can verify.