1352330 - Telemetry - Record one for each non-empty field at submission time (keyed by fieldName)

Status: VERIFIED FIXED

(Toolkit :: Form Autofill, enhancement, P3)

Description

[Steve Chung [:steveck]]

Product Owner should be able to see the number of the fields that are:
- Not autofilled
- Autofilled without edited
- Autofilled anf modified
With these number, we should be able to analysis whether our heuristic algorithm is accurate or not while filling the profiles.

Comment 1

[Steve Chung [:steveck]]

Since it's not the original PM request, I moved it out of m4 for now. But it might still be helpful for engineering(such as the accuracy of the heuristic). I'll discuss with Sean for more details.

Comment 2

[Steve Chung [:steveck]]

After discussion with Sean, we would prefer to acquire more information from this probe instead of numbers only. We think that knowing number might be interesting, but we could not take any further action from these numbers. For example, we might be able to know that address field's modified rate is higher then others, but we could not know how to improve it from the rate. Hi MattN, is there any other methods that we could record more information in the probe(leverage heart beat or customized ping)?

Comment 3

[Matthew N. [:MattN]]

What we would be able to do is compare the metrics across versions to understand if we regressed our success rate. We wouldn't know how we regressed but we would know that we did. We need would need differential privacy or opt-in UI for users to submit domains/URLs but both of those are complicated issues. I still think this probe should be added.

Comment 5

[Zibi Braniecki [:zbraniecki][:gandalf]]

It would be very valuable for us to have this telemetry be keyed on locale to be able to monitor regional differences and spot markets at which our heuristics backfire.

Comment 6

[Matthew N. [:MattN]]

All telemetry has the OS and browser locale. If you mean the page locale then that's more complicated and expensive.

Comment 7

[Zibi Braniecki [:zbraniecki][:gandalf]]

All telemetry has the OS and browser locale. If you mean the page locale then that's more complicated and expensive.

No, I meant browser locale chain (so, locales, not a singular locale, I hope that's what you mean).

Comment 8

[Matthew N. [:MattN]]

(In reply to Zibi Braniecki [:zbraniecki][:gandalf] from comment #7)

All telemetry has the OS and browser locale. If you mean the page locale then that's more complicated and expensive.

No, I meant browser locale chain (so, locales, not a singular locale, I hope that's what you mean).

Ok, glad you don't mean page locale. It looks like many of the lists are in the environment data. I guess check if that is sufficient: about:telemetry#environment-data-tab

Comment 9

[Jim Mathies [:jimm]]

Not required for the mvp.

Comment 10

[Zibi Braniecki [:zbraniecki][:gandalf]]

Leif - can you confirm if I can add this as three extra keys to cc_form/filled event?

{
  cardholder_name: (filled|not_filled|unavailable),
  card_number: (filled|not_filled|unavailable),
  expr_date: (filled|not_filled|unavailable),

  // New extra keys
  fields_not_autofilled: int,
  fields_autofilled_and_modified: int,
  fields_autofilled: int,
}

Comment 11

[Leif Oines [:loines]]

Yes, if those are in addition to the first three listed, I'm good with that.

Comment 12

[Zibi Braniecki [:zbraniecki][:gandalf]]

Attachment: Bug 1352330 - Credit Card Telemetry for number of fields detected/autofilled/modified.

Comment 13

[Zibi Braniecki [:zbraniecki][:gandalf]]

Leif,

After some mingling with the data I came up with the following extra keys:

{
      fields_not_auto: Number of fields in a `submitted` event that were not autofilled
      fields_auto: Number of fields in the `submitted` event that were autofilled
      fields_modified: Number of fields in a `submitted` event that were autofilled and later modified
}

This is in line with this bug's comment 0.

Few notes I want to verify with you:

1. The values of the extra keys in events must be Strings. That means that what is sent is:

    [
      18286,
      "creditcard",
      "submitted",
      "cc_form",
      "{32e2efd2-d641-4677-b1d9-379f0bcf19eb}",
      {
        "fields_not_auto": "3",
        "fields_auto": "5",
        "fields_modified": "1"
      }
    ]

Is that usable in telemetry?

2. When we calculate fields, we take a total count of fields in a form and deduct the number of autofilled fields), that means that <button type="submit"/> may be included in the totalCount so a form with 4 inputs and two buttons will have 6 total fields. I hope that's ok for now, and we can tailor the types of fields if we need it.
3. The keys are per record, and there may be multiple records in a single form in the future (say, address+credit card records), but the data is "fields_not_auto" is calculated by taking a total number of fields in the form, and then removing the number of fields this record autofilled.

For now, it will work - we expect to have only one record in a form, but in the future you may have a form with 12 fields, a record with 4 fields (CC) and another record with 5 fields (Address), and in submission you will get two "submitted" events, one per record:

[
  // CC record
  {
      fields_not_auto: 8,
      fields_auto: 4,
      fields_modified: 0,
  },
  // Address record
  {
      fields_not_auto: 7,
      fields_auto: 5,
      fields_modified: 0,
  }
]

Comment 14

[Leif Oines [:loines]]

1. Strings are OK, they can be cast to int at analysis time.
2. I think this is OK, at least for now. We should probably document that fields_not_auto can count buttons, perhaps in events.yaml.
3. Curious, is this because in the future the mechanism for address and CC autofill will be merged? You're right that I think its OK for now since we know this telemetry only covers creditcards, but we should think about how we can prepare and/or future proof it so we at least know when it will begin to report the behavior of address autofill as well. Or are you saying that we will already be able to tell from the event what type of record was filled?

Comment 15

[Zibi Braniecki [:zbraniecki][:gandalf]]

Curious, is this because in the future the mechanism for address and CC autofill will be merged?

It's mostly because of how the code categorizes a form - it divides it into sections which are of certain type (cc, address, unknown).
In result, a single form with 25 fields may have multiple sections of different kinds.

When we submit that telemetry, we submit it per-section, not per-form, so there's no event per-form, just per-section. And in result, when I submit an event because a section is being submitted, I can only answer with information about this section and its relation to the whole form.

I could, in theory, add more telemetry that would do things like submit event per-form and then accumulated data from all its sections, or in section submission event I could add accumulated per-form information, but both are quirky.

I guess fine tuning of (3) will require us to better understand what questions we want to ask of data here. Do we want to know how many fields of a form were autofilled from our CC mechanism? Or do we want to understand how often we autofill into big forms or small forms?
Depending on those answers we may want to slice the sections/forms differently, but that will require probe fine-tuning.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c70f82d54c77
Credit Card Telemetry for number of fields detected/autofilled/modified. r=abr

Comment 17

[Zibi Braniecki [:zbraniecki][:gandalf]]

Comment on attachment 9166683 [details]
Bug 1352330 - Credit Card Telemetry for number of fields detected/autofilled/modified.

Beta/Release Uplift Approval Request

User impact if declined: None
Is this code covered by automated tests?: Yes
Has the fix been verified in Nightly?: Yes
Needs manual test from QE?: No
If yes, steps to reproduce:
List of other uplifts needed:
Risk to taking this patch: Low
Why is the change risky/not risky? (and alternatives if risky): It only adds telemetry
String changes made/needed: None

Comment 18

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/c70f82d54c77

Comment 19

[Julien Cristau [:jcristau]]

Comment on attachment 9166683 [details]
Bug 1352330 - Credit Card Telemetry for number of fields detected/autofilled/modified.

approved for 80.0b5

Comment 20

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/6b7318cdd54a

Comment 21

[Cosmin Muntean [:cmuntean], Ecosystem QA]

I have verified the telemetry implemented in this bug on Firefox 80.0b6 (Build ID: 20200807195315) and the latest Nightly 81.0a1 build (Build ID: 20200809213940) on Windows 10 x64, Mac 10.14.6, Linux Mint 19.2 and Ubuntu 20.04 x64.
In order to verify this we have used the Credit Card Autofill Metrics Spec document.

• The submit event contains the {fields_not_auto: "", fields_auto: "", fields_modified: ""} extra values.
• However while verifying this event I have encountered an issue with the "fields_modified" and logged it in Bug 1657909.