1412358 - Add telemetry probes for actions on accessibility indicator button and accessibility privacy preference checkbox.

Status: RESOLVED FIXED

(Firefox :: General, defect, P1)

Description

[Yura Zenevich [:yzen] (please use "needinfo")]

During the Firefox 57 developer sprint we added two user facing accessibility related features to Firefox front end:

* accessibility indicator for when accessibility services are activated in Firefox (see bug 1383051)

* privacy preference for accessibility services that allows users to disable accessibility when might be used by tools other than assistive technologies (see bug 1384567)

The privacy preference is going to be available to all users as part of Firefox 57. The indicator was decided to be preffed off for the 57 release because there is a concern about it (and the relevant SUMO page - https://support.mozilla.org/en-US/kb/accessibility-services that it links to) being confusing to the user. 

We are planning to run a shield study in Firefox 57 that will help us decide if the indicator has a positive effect on our accessibility and non-accessibility users. For that we would like to add telemetry probes to be able to track the number of users who clicked on the indicator and who checked/unchecked the privacy preference. And, as a consequence, how these actions affect user retention and satisfaction.

Comment 1

[Yura Zenevich [:yzen] (please use "needinfo")]

Attachment: 1412358 patch

Comment 2

[Yura Zenevich [:yzen] (please use "needinfo")]

Comment on attachment 8922925 [details] [diff] [review]
1412358 patch

Background
See comment 0.

Data Collected
When the user clicks or presses Enter/Space on the accessibility indicator button, we open a new tab with the SUMO page about accessibility services - https://support.mozilla.org/en-US/kb/accessibility-services. We would like to record when that event happens, or in other words, if the user opened a support page through our accessibility indicator button.

We would like to record when the user checks a checkbox in Privacy & Security section of Firefox preferences for preventing accessibility services from accessing Firefox.

Purpose of Collection
These telemetry probes are necessary for our upcoming shield study (bug 1411590). We would like to know if showing the accessibility services indicator and checking the privacy checkbox have positive effect on user satisfaction and retention. 
We should also be able to see, through the study, if the indicator is confusing and/or the support page needs additional refinement.

Comment 3

[Jim Mathies [:jimm]]

Comment on attachment 8922925 [details] [diff] [review]
1412358 patch

Review of attachment 8922925 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 4

[Yura Zenevich [:yzen] (please use "needinfo")]

Data Review request:

https://gist.github.com/yzen/adc5435d9b62dfa454effdfbbcb9cffd

Comment 5

[[e-mail]]

Comment on attachment 8922925 [details] [diff] [review]
1412358 patch

Review of attachment 8922925 [details] [diff] [review]:
-----------------------------------------------------------------

Providing answers to the review questions here: https://github.com/mozilla/data-review/blob/master/review.md
1. Yes, these are telemetry probes and will have the standard Telemetry documentation.
2. Yes, users can control their collection via the telemetry preference setting.
3. No, this is temporary.
4. Type 2 measurement.
5. Default on in release, all locales, all countries (I believe).
6. No new identifiers.
7. Covered under policy.
8. No need for future check-in.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/905db84ed172
add telemetry probes for actions on accessibility indicator button and accessibility privacy preference checkbox. r=jimm

Comment 7

[Yura Zenevich [:yzen] (please use "needinfo")]

Comment on attachment 8922925 [details] [diff] [review]
1412358 patch

Initially we were going to land accessibility indicator by default to all users in Firefox 57. However, with concerns regarding potentially confusing our users, we opted for making the indicator preffed off by default. Instead, we plan to run a Shield Study for different user cohorts with the indicator enabled to see if it has a positive effect on the user base. This is the only remaining blocking bug for the Shield Study.

[Feature/Bug causing the regression]: Not a regression.
[User impact if declined]: We would not be able to run a Shield Study in Firefox 57. We believe the indicator will have a positive impact on user retention and satisfaction  but we need to run the study first to prove this hypothesis.
[Is this code covered by automated tests?]: No, adding simple telemetry probe.
[Has the fix been verified in Nightly?]: Not yet
[Needs manual test from QE? If yes, steps to reproduce]: No
[List of other uplifts needed for the feature/fix]: None
[Is the change risky?]: No
[Why is the change risky/not risky?]: This patch is just a simple telemetry probe change.
[String changes made/needed]: None

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/905db84ed172

Comment 9

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8922925 [details] [diff] [review]
1412358 patch

This telemetry probe is needed to get value out of the planned a11y indicator shield study during release 57 cycle. Beta57+

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/d57f1c698b4e