Closed Bug 990199 Opened 6 years ago Closed 3 years ago

[Form Autofill] Collect information on how many people opt-out of Form Autofill

Categories

(Toolkit :: Form Manager, defect, P3)

defect

Tracking

()

RESOLVED FIXED
mozilla56
Tracking Status
firefox56 --- fixed

People

(Reporter: sevaan, Assigned: steveck)

References

(Blocks 1 open bug)

Details

(Whiteboard: [story][form autofill:M4][autofill-metrics])

User Story

As a Product Owner, I want to be able to see data on how many people opt-out of Form Autofill so that I can better understand the utility of the feature.

Acceptance Criteria:

- I can view a report that show me how many people have chosen to disable or opt-out of being offered form autofill though the following ways:
--- Choosing the “don’t ever save or ask me again” advanced option from the doorhanger 
--- Choosing the “don’t save this profile” advanced option from the doorhanger
--- Choosing to disable autofill from the browser preferences

Attachments

(1 file)

As a Product Owner I want to be able to see data on how many people opt-out of Automatic Translation so that I can better understand the utility of form autofill.

Acceptance Criteria:

- I can view a report that show me how many people have chosen to disable or opt-out of being offered form autofill though the following ways:
--- Choosing the “don’t ever save or ask me again” advanced option from the doorhanger 
--- Choosing the “don’t save this profile” advanced option from the doorhanger
--- Choosing to disable autofill from the browser preferences
Whiteboard: [story] [form autofill]
Depends on: 990234
Summary: [Form Autofill] Collect information on how many people opt-out of Automatic Translation → [Form Autofill] Collect information on how many people opt-out of Form Autofill
Incorrect summary above! This is not an Automatic Translation project:

As a Product Owner I want to be able to see data on how many people opt-out of form autofill so that I can better understand the utility of the feature.

Acceptance Criteria:

- I can view a report that show me how many people have chosen to disable or opt-out of being offered form autofill though the following ways:
--- Choosing the “don’t ever save or ask me again” advanced option from the doorhanger 
--- Choosing the “don’t save this profile” advanced option from the doorhanger
--- Choosing to disable autofill from the browser preferences
No longer blocks: fxdesktopbacklog
Flags: firefox-backlog+
User Story: (updated)
Component: General → Form Manager
Product: Firefox → Toolkit
Whiteboard: [story] [form autofill] → [story] [form autofill][autofill-metrics]
We could apply telemetry scalar that sets different labels for different opt-out situations, if we want to leverage main ping.
Whiteboard: [story] [form autofill][autofill-metrics] → [story][form autofill:M2][autofill-metrics]
Whiteboard: [story][form autofill:M2][autofill-metrics] → [story][form autofill:M3][autofill-metrics]
Comment on attachment 8861790 [details]
Bug 990199 - Record address autofill enabled state.

Hi Matt,
It's a WIP about how I might collect data though scalars in this metrics. I applied the string metrics for 3 different states(enabled/disabled-from-pref/disabled-from-doorhanger) and the it'll be set while addon startup(storage initialization) and changed in pref/doorhanger. Since Joe wants to know the state of the feature be in used per subsession instead of times, I think it could work  if we always have an string value set to the scalars. In order to keep the state after browser closed, I stored the profile storage and there might be other better solution for keeping the state(adding another pref sounds like another abuse of prefs).
Attachment #8861790 - Flags: feedback?(MattN+bmo)
Assignee: nobody → schung
Status: NEW → ASSIGNED
OS: Mac OS X → All
Priority: -- → P3
Hardware: x86 → All
Georg suggested using the Telemetry Environment for recording the enabled/disabled state: https://dxr.mozilla.org/mozilla-central/rev/5278e2a35fc8f2be390243db1e62858bf0982055/toolkit/components/telemetry/TelemetryEnvironment.jsm#158

We would still want to record how it was disabled separately.
Hi Georg, thanks for the advice! Is there any way we can see the dashboard about the Telemetry Environment or set it as the filter in the dashboard? Or we could only leverage Telemetry Environment data by building the custom ping and render the dashboard by by ourselves?
Flags: needinfo?(gfritzsche)
(In reply to Steve Chung [:steveck] from comment #6)
> Hi Georg, thanks for the advice! Is there any way we can see the dashboard
> about the Telemetry Environment or set it as the filter in the dashboard? Or
> we could only leverage Telemetry Environment data by building the custom
> ping and render the dashboard by by ourselves?

I believe it's not on the t.m.o dashboards but it's easy to query for it on sql.telemetry.mozilla.org and make a graph. Dashboards for prefs in the environment seem like a reasonable request though.
Right, use re:dash (sql.telemetry.mozilla.org), with e.g. the longitudinal data set. [1][2]

Bug 1336977 is how we plan to get to showing the environment data in the TMO dashboard; that is a longer-term project though.

1: https://github.com/mozilla/telemetry-batch-view/blob/master/docs/choosing_a_dataset.md#longitudinal
2: https://github.com/mozilla/telemetry-batch-view/blob/master/docs/longitudinal_examples.md
Flags: needinfo?(gfritzsche)
Comment on attachment 8861790 [details]
Bug 990199 - Record address autofill enabled state.

https://reviewboard.mozilla.org/r/133790/#review140406

::: toolkit/components/telemetry/Scalars.yaml:584
(Diff revision 2)
>      record_in_processes:
>        - 'main'
> +
> +# The following section contains the form autofill related scalars.
> +formautofill:
> +  address_opt_out_state:

Maybe I should give it a better naming since this scalar is for recording a boolean that whether the autofill is opt-out through preference panel.
Hi,
I add telemetryEnvironment and scalar in the patch, but I would like to know if the scalar here is a proper option. Joe's requirement is to know the daily users who disabled the feature(and how did they disable). We'll apply telemetryEnvironment for the feature disable part, but I'm sure that using the scalar would keep the state about how user disabling the feature. IIRC the value would be sent session by session, so using scalar here could only record the count in a session instead of keeping the value permanently. Could we create a chart in redash by this scalar(or maybe event), or is there a better way to keep a value like telemetryEnvironment?
Flags: needinfo?(gfritzsche)
Flags: needinfo?(MattN+bmo)
Is tracking enabled/disabled user counts or ratios sufficient for daily values?
Does it add value to know on a daily basis *how* users enabled/disabled?

Not graphing that on a daily basis makes the querying "how users enabled/disabled" more easy/performant:
- (grab only subsessions from say the last week/month/...)
- grab any subsession that has an "address_opt_out_state"
- (reduce to per client values if needed)
- produce totals/ratios/graph

But you can just as well write a query to break this down to daily values if needed.
Flags: needinfo?(gfritzsche)
Comment on attachment 8861790 [details]
Bug 990199 - Record address autofill enabled state.

https://reviewboard.mozilla.org/r/133790/#review141364

::: toolkit/components/telemetry/Scalars.yaml:583
(Diff revision 2)
> +formautofill:
> +  address_opt_out_state:
> +    bug_numbers:
> +      - 990199
> +    description: State of feature is enabled or disabled from doorhanger/preferences.
> +    expires: never
> +    kind: boolean
> +    notification_emails:
> +      - autofill@lists.mozilla.org
> +    release_channel_collection: opt-out
> +    record_in_processes:
> +      - 'main'

How about a string scalar where the string is either "preferences" or "doorhanger". That gives us the flexibility to add a third location (unlike booleans) and doesn't require memorizing the meaning of true vs. false.

I think a name of "disable_ui_used" under the "formautofill.addresses" group may be clearer with a description like "Indicates the UI that was used to disable address autofill"

::: toolkit/components/telemetry/TelemetryEnvironment.jsm:173
(Diff revision 2)
>    ["app.update.url", {what: RECORD_PREF_VALUE}],
>    ["browser.cache.disk.enable", {what: RECORD_PREF_VALUE}],
>    ["browser.cache.disk.capacity", {what: RECORD_PREF_VALUE}],
>    ["browser.cache.memory.enable", {what: RECORD_PREF_VALUE}],
>    ["browser.cache.offline.enable", {what: RECORD_PREF_VALUE}],
> +  ["browser.formautofill.enabled" , {what: RECORD_PREF_VALUE}],

We should really rename this pref in a separate commit before landing this since it doesn't indicate it's for address autofill. e.g. browser.formautofill.addresses.enabled
(In reply to Georg Fritzsche [:gfritzsche] from comment #12)
> Is tracking enabled/disabled user counts or ratios sufficient for daily
> values?
> Does it add value to know on a daily basis *how* users enabled/disabled?

Good question, we want to know how many users who decided to disable the feature has tried the feature.

If the users disables from the doorhanger(It only shows at the first time when user submit a form), it may imply that user doesn't want this feature at the very beginning; if users disables the feature from preference, it's very likely that they tried autofill but they still prefer to disable this feature. We might need to provide better use experience if the number/ratio of disabling from pref is high.

Ni? Joe since he might be able to give additional explanation for the purpose.
Flags: needinfo?(jcheng)
I'm not 100% if you got the answer you want from Georg already but recording only within the session where the user actually switches from enabled to disabled seems sufficient for what we need as you can easily query previous session with the longitudinal dataset.
Flags: needinfo?(MattN+bmo)
Depends on: 1303510
Comment on attachment 8861790 [details]
Bug 990199 - Record address autofill enabled state.

https://reviewboard.mozilla.org/r/133790/#review146898

::: browser/extensions/formautofill/FormAutofillParent.jsm:296
(Diff revision 3)
>            Services.prefs.setBoolPref(ENABLED_PREF, false);
>            this.profileStorage.addresses.getAll().forEach((record) => {
>              this.profileStorage.addresses.remove(record.guid);
>            });
> +
> +          Services.telemetry.scalarSet("formautofill.address.disable_ui_used",

Not sure if we need to verify the scalar result in browser test because I barely found people add test for scalars(but some tests for histogram)... Shall we write test for it?
Comment on attachment 8861790 [details]
Bug 990199 - Record address autofill enabled state.

https://reviewboard.mozilla.org/r/133790/#review148600

::: commit-message-73752:1
(Diff revision 3)
> +Bug 990199 - Part 1: Add scalars for formautofill state in preference. r?MattN

Nit: Bug 990199 - Record form autofill enabled state in scalars. r?MattN

::: toolkit/components/telemetry/Scalars.yaml:462
(Diff revision 3)
> +# The following section contains the form autofill related scalars.
> +formautofill.address:
> +  disable_ui_used:
> +    bug_numbers:
> +      - 990199

Please request data review before landing. I think the code aspects are fine other than nits. https://wiki.mozilla.org/Firefox/Data_Collection#Requesting_Approval

::: toolkit/components/telemetry/Scalars.yaml:467
(Diff revision 3)
> +# The following section contains the form autofill related scalars.
> +formautofill.address:
> +  disable_ui_used:
> +    bug_numbers:
> +      - 990199
> +    description: State of feature is enabled or disabled from doorhanger/preferences.

Nit: s/State of// since it's only recording an "event"-like thing

::: toolkit/components/telemetry/TelemetryEnvironment.jsm:197
(Diff revision 3)
>    ["extensions.allow-non-mpc-extensions", {what: RECORD_PREF_VALUE}],
>    ["extensions.autoDisableScopes", {what: RECORD_PREF_VALUE}],
>    ["extensions.enabledScopes", {what: RECORD_PREF_VALUE}],
>    ["extensions.blocklist.enabled", {what: RECORD_PREF_VALUE}],
>    ["extensions.blocklist.url", {what: RECORD_PREF_VALUE}],
> +  ["extensions.formautofill.addresses.enabled" , {what: RECORD_PREF_VALUE}],

Nit: there is an extra space before the comma
Attachment #8861790 - Flags: review?(MattN+bmo) → review+
Comment on attachment 8861790 [details]
Bug 990199 - Record address autofill enabled state.

https://reviewboard.mozilla.org/r/133790/#review146898

> Not sure if we need to verify the scalar result in browser test because I barely found people add test for scalars(but some tests for histogram)... Shall we write test for it?

I don't think it's necessary. We can do a manual verification.
Comment on attachment 8861790 [details]
Bug 990199 - Record address autofill enabled state.

Hi Georg, I think you're the best candidate for data review since you might know the purpose of this probe already. Please let me know if I should ping someone else for data review, thanks!
Attachment #8861790 - Flags: feedback?(gfritzsche)
Comment on attachment 8861790 [details]
Bug 990199 - Record address autofill enabled state.

Georg isn't a data collection reviewer according to https://wiki.mozilla.org/Firefox/Data_Collection so redirecting to Chenxia.
Attachment #8861790 - Flags: feedback?(gfritzsche) → feedback?(liuche)
Whiteboard: [story][form autofill:M3][autofill-metrics] → [story][form autofill:M4][autofill-metrics]
Since there won't have a doorhanger for user to disable autofill, this metrics could be simplified into 1 line insertion that adds the enabled flag in telemetry environment.

Hi Benjamin, may I know who could review the new metrics that added in the system addon?
Flags: needinfo?(benjamin)
Sorry for not being active on this bug, for some reason I'm not seeing the f? request.

I'm assuming that extensions.formautofill.addresses.enabled controls whether Firefox autofills for (mailing) addresses, which is fine - not personally identifiable, just a preference that we're collecting for usage. However, I'm not seeing clear documentation for it anywhere after a quick dxr search, so can you document specifically what this is collecting somewhere it's being defined?

Not sure if TelemetryEnvironment data collection/documentation protocol is different, please let me know if it is.
Hi Chenxia, thanks for the reply and I'm not sure why the feedback didn't work as expected either... Sorry about that :/

Is that sufficient if I add some comment about the pref flag, or I should add more explanation in environment.rst? I could not find any document about the telemetry environment data review process, and I saw some people added flag in telemetry environment without any information. Please let me know the data team's preference and I'll update the comment/document ASAP, thanks!
Flags: needinfo?(benjamin) → needinfo?(liuche)
Comment on attachment 8861790 [details]
Bug 990199 - Record address autofill enabled state.

Hi Benjamin, I'm not quite sure about how to add the documentation for new flag in telemetry environment, so I only add a comment for this flag at first. Please let me know if it doesn't meet the data team's requirement, thanks.
Attachment #8861790 - Flags: feedback?(benjamin)
Comment on attachment 8861790 [details]
Bug 990199 - Record address autofill enabled state.

https://reviewboard.mozilla.org/r/133790/#review154650

data-r=me

We should probably document the set of prefs that are included in the environment, but we don't have a scheme for that currently so this is fine.
Attachment #8861790 - Flags: review+
Flags: needinfo?(liuche)
Attachment #8861790 - Flags: feedback?(benjamin)
Pushed by mozilla@noorenberghe.ca:
https://hg.mozilla.org/integration/autoland/rev/0c3542cbd56f
Record address autofill enabled state. r=bsmedberg,MattN
https://hg.mozilla.org/mozilla-central/rev/0c3542cbd56f
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
Flags: needinfo?(jcheng) → needinfo?(chsiang)
clearing the ni as the bug is resolved already. :)
Flags: needinfo?(chsiang)
You need to log in before you can comment on or make changes to this bug.