[Form Autofill] Collect information about the count of autofill being applied in different scenario

RESOLVED FIXED in Firefox 56

Status

()

defect
RESOLVED FIXED
5 years ago
2 years ago

People

(Reporter: sevaan, Assigned: steveck)

Tracking

(Blocks 2 bugs)

unspecified
mozilla56
Points:
1
Dependency tree / graph
Bug Flags:
firefox-backlog +

Firefox Tracking Flags

(firefox56 fixed)

Details

(Whiteboard: [form autofill:M4][autofill-metrics][ETA:7/21])

User Story

As a Product Owner, I want to be able to see data on how many people select an autofill profile so that I can better understand the efficacy of form autofill.

Acceptance Criteria:

- I can view a report that shows me how many times users have selected to autofill a form using a stored profile.

Attachments

(2 attachments)

Reporter

Description

5 years ago
As a Product Owner I want to be able to see data on how many people select an autofill profile so that I can better understand the efficacy of form autofill.

Acceptance Criteria:

- I can view a report that shows me how many times users have selected to autofill a form using a stored profile.
Whiteboard: [story] [form autofill]
Depends on: 990236
No longer blocks: fxdesktopbacklog
Flags: firefox-backlog+
Hey Sevaan,

This seems very similar to bug 990204 so clarification on the distinction would be good (or duping them otherwise). Using an auto-fill profile is basically the same thing as auto-filling a form with the profile. Please clarify how the data will be used and/or what the calculation would be.

I suspect some of these pieces of telemetry would prefer knowing a proportion rather than "how many". e.g. what proportion of forms that a user sees do they use autofill on rather than data that a given user autofilled 5 forms since that tells you nothing about their session length and how many forms they saw.

Thanks
Component: General → Form Manager
Flags: needinfo?(sfranks)
OS: Mac OS X → All
Product: Firefox → Toolkit
Hardware: x86 → All
See Also: → 990204
Whiteboard: [story] [form autofill] → [form autofill] p=1
Reporter

Comment 2

5 years ago
Hey Matt,

You're right, the bugs are very similar, but I think the distinction we wanted to look at was:

- Bug 990200: Measure the number of times a user actually autofills a form.
- Bug 990204: Count the average number of profiles a user accrues and uses

My thinking regarding the former (Bug 990200) was just to see whether users were actually engaging with the feature. I was thinking a % of users who autofill forms as oppose to a straight up number of times a form was autofilled. But your note on proportion makes a lot more sense and does seem more useful. We will need to make some distinction, though, regarding any old form, and "Profile-worthy" forms (i.e. forms with name/address fields).

Regarding the latter (Bug 990204) we wanted to measure the average number of profiles commonly used to fill out forms. For example, each time you fill out a form with slightly different information, a new profile is created. And I've noticed Google Chrome keeps A LOT of different profiles, but I probably only really use 1-2 of them. It would be handy to know how many profiles users have on average, and how many they are actually using, and in the future use this information to a) better determine what information gets saved, and b) possibly introduce a data clean-up feature that can excise any useless information we are storing, like addresses you may have entered in once. Or maybe there is a way to ask the user "Hey, this profile is very similar to another profile you have entered, but the phone number is different, would you like to update the old profile or create a new one"?

Just some of my initial thoughts, but I am opened to much more experienced opinions regarding data collection here. I just don't want to miss anything where in the future we'll go "Darn, I wish we were measuring X, so we would know Y".
Flags: needinfo?(sfranks)
Points: --- → 1
Whiteboard: [form autofill] p=1 → [form autofill]
User Story: (updated)
Whiteboard: [form autofill] → [form autofill][autofill-metrics]
Assignee

Comment 3

2 years ago
Hi Joe, will we still need this metric, or replace this by bug 1341569? IIRC we would prefer to just record the form filling time during the work weeks.
Flags: needinfo?(jcheng)
Assignee

Comment 4

2 years ago
Sorry, comment 3 should be related to bug 990201 actually.
Flags: needinfo?(jcheng)
Assignee

Comment 5

2 years ago
We could apply telemetry categorical histogram to add the times that applies autofill or not while submitting the form.
Whiteboard: [form autofill][autofill-metrics] → [form autofill:M2][autofill-metrics]
Assignee

Comment 6

2 years ago
(In reply to Steve Chung [:steveck] from comment #5)
> We could apply telemetry categorical histogram to add the times that applies
> autofill or not while submitting the form.

categorical histogram -> keyed scalar
Whiteboard: [form autofill:M2][autofill-metrics] → [form autofill:M3][autofill-metrics]
Assignee

Comment 7

2 years ago
After discussion with Joe, we'll record the count of autofill feature being applied instead of the number of users. The results could be classified into 3 different categories:
- Manual input
- Autofill the form without any change
- Autofill the form but with modified results

We'll ignore the the scenario if user disable the function or function enabled but no profile saved, since user has no chance to see/apply the feature in these 2 cases and we might add too many noise in the "Manual input" category.
Summary: [Form Autofill] Collect information on how many people autofill forms → [Form Autofill] Collect information about the count of autofill being applied in different scenario
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Assignee

Updated

2 years ago
Assignee: nobody → schung
Whiteboard: [form autofill:M3][autofill-metrics] → [form autofill:M3][autofill-metrics][ETA:612]
Assignee

Updated

2 years ago
Whiteboard: [form autofill:M3][autofill-metrics][ETA:612] → [form autofill:M3][autofill-metrics]
Whiteboard: [form autofill:M3][autofill-metrics] → [form autofill:M4][autofill-metrics]
Comment on attachment 8865769 [details]
Bug 990200 - Part 1: Add 3 filling type scalar for the autofill type count.

https://reviewboard.mozilla.org/r/137390/#review158958

::: toolkit/components/telemetry/Scalars.yaml:463
(Diff revision 2)
>      release_channel_collection: opt-out
>      record_in_processes:
>        - 'main'
>  
> +# The following section contains the form autofill related scalars.
> +formautofill.address:

Nit: formautofill.addresses since we seem to always use the plural form (prefs, storage, etc.)

::: toolkit/components/telemetry/Scalars.yaml:464
(Diff revision 2)
>      record_in_processes:
>        - 'main'
>  
> +# The following section contains the form autofill related scalars.
> +formautofill.address:
> +  filling_type:

Nit: fill_type

::: toolkit/components/telemetry/Scalars.yaml:467
(Diff revision 2)
> +# The following section contains the form autofill related scalars.
> +formautofill.address:
> +  filling_type:
> +    bug_numbers:
> +      - 990200
> +    description: Number of counts for different form filling type.

Count at submission time whether the submitted form was autofilled.
Attachment #8865769 - Flags: review?(MattN+bmo) → review+
Comment on attachment 8873685 [details]
Bug 990200 - Part 2: Set probes in the form submission part.

https://reviewboard.mozilla.org/r/145074/#review158960

r=me after rebasing
Attachment #8873685 - Flags: review?(MattN+bmo) → review+
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Assignee

Comment 15

2 years ago
Comment on attachment 8865769 [details]
Bug 990200 - Part 1: Add 3 filling type scalar for the autofill type count.

Hi Benjamin, do you think using string type scalar here is fine? Cause I didn't find anyone use string scalar before except the test.
Attachment #8865769 - Flags: feedback?(benjamin)

Comment 16

2 years ago
mozreview-review
Comment on attachment 8865769 [details]
Bug 990200 - Part 1: Add 3 filling type scalar for the autofill type count.

https://reviewboard.mozilla.org/r/137390/#review160286

::: toolkit/components/telemetry/Scalars.yaml:732
(Diff revision 3)
> +# The following section contains the form autofill related scalars.
> +formautofill.addresses:
> +  fill_type:
> +    bug_numbers:
> +      - 990200
> +    description: Count at submission time whether the submitted form was autofilled.

This is marked as a keyed int scalar but the description doesn't document what the key values are. Please document carefully what key values are.

It wasn't clear reading the bug what the values could be either, but if it's just the three categories:

- Manual input
- Autofill the form without any change
- Autofill the form but with modified results

Might it make more sense for these to just be three different scalars? That would be more effeicient to send, store and query in main_summary.

e.g.
 formautofill.addresses:
   fill_type_manual:
     ...
   fill_type_auto:
     ...
   fill_type_auto_modified:
     ...
Attachment #8865769 - Flags: review-

Updated

2 years ago
Attachment #8865769 - Flags: feedback?(benjamin)
Assignee

Updated

2 years ago
Whiteboard: [form autofill:M4][autofill-metrics] → [form autofill:M4][autofill-metrics][ETA:7/21]
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 19

2 years ago
mozreview-review
Comment on attachment 8865769 [details]
Bug 990200 - Part 1: Add 3 filling type scalar for the autofill type count.

https://reviewboard.mozilla.org/r/137390/#review163544

data-r=me when you add a real person to the notification_emails.

I read through the implementation and I didn't see an automated test that ensures the telemetry is collected properly. Either here or as a followup bug it's important to have an automated test for expires: never telemetry, because it's really easy to break telemetry during refactorings without noticing and without QA picking up the breakage.

::: toolkit/components/telemetry/Scalars.yaml:736
(Diff revision 4)
> +      - 990200
> +    description: Count at submission time whether the submitted form was filled manually.
> +    expires: never
> +    kind: int
> +    notification_emails:
> +      - autofill@lists.mozilla.org

It is perfectly fine to have a list alias here, but we are asking that you also have the DRI listed here, which I assume is sevaan? Whoever is responsible for making use of this data/watching a dashboard. This is especially important for expires: never metrics.

ditto for the other two

::: toolkit/components/telemetry/Scalars.yaml:739
(Diff revision 4)
> +    kind: int
> +    notification_emails:
> +      - autofill@lists.mozilla.org
> +    release_channel_collection: opt-out
> +    record_in_processes:
> +      - 'main'

Technical not data-review question: Here and below, are you sure that this will be main-process only? Form submission normally takes place in content processes.
Attachment #8865769 - Flags: review?(benjamin) → review+
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Assignee

Comment 22

2 years ago
mozreview-review-reply
Comment on attachment 8865769 [details]
Bug 990200 - Part 1: Add 3 filling type scalar for the autofill type count.

https://reviewboard.mozilla.org/r/137390/#review163544

> Technical not data-review question: Here and below, are you sure that this will be main-process only? Form submission normally takes place in content processes.

Thanks for the reminder! Yes submission normally takes place in content processes, but we'll relay the submission from content to parent process since it'll need to handle storage and doorhanger(it's easier to handle these tasks in parent process). You can see the probes I added is in FormAutofillParent.jsm, which means it's running on parent.
Assignee

Updated

2 years ago
Blocks: 1382143
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Assignee

Updated

2 years ago
Keywords: checkin-needed

Comment 25

2 years ago
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/6f8553a4b235
Part 1: Add 3 filling type scalar for the autofill type count. r=bsmedberg,MattN
https://hg.mozilla.org/integration/autoland/rev/1ca6ba47a88a
Part 2: Set probes in the form submission part. r=MattN
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/6f8553a4b235
https://hg.mozilla.org/mozilla-central/rev/1ca6ba47a88a
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
Assignee

Updated

2 years ago
Blocks: 1387316
Assignee

Updated

2 years ago
Duplicate of this bug: 990207
You need to log in before you can comment on or make changes to this bug.