1412603 - [Shield] Pref Flip Study: accessibility visual indicator and permission preference

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Yura Zenevich [:yzen] (please use "needinfo")]

Basic description of experiment:
Create 6 different cohorts that have the same user characteristics (representative sample of the release population). Proposed cohorts in detail:
 * New users that are a11y users (A11Y_INSTANTIATED_FLAG telemetry) and have default preference (indicator disabled - default)
 * Existing users that are a11y users (A11Y_INSTANTIATED_FLAG telemetry) and have 
default preference (indicator disabled - default)
 * New user that are a11y users (A11Y_INSTANTIATED_FLAG telemetry) and have preference changed (indicator enabled)
 * Existing users that are a11y users (A11Y_INSTANTIATED_FLAG telemetry) and have preference changed (indicator enabled)
 * New users that are not a11y users (indicator disabled - default)
 * Existing users that are not a11y users (indicator disabled - default)


What is the preference we will be changing?
'accessibility.indicator.enabled'


What are the branches of the study and what values should each branch be set to?
(indicator enabled) groups above: true
(indicator disabled - default) groups above: false


What percentage of users do you want in each branch?
TBC with :joy


What Channels and locales do you intend to ship to?
Release channel, all locales, 57 target 


What is your intended go live date and how long will the study run?
TBC with :joy


Are there specific criteria for participants?
No, both new and existing users


What is the main effect you are looking for and what data will you use to make these decisions?
 * Session length
 * Subsession length
 * 2 week retention
 * jank / hangs / performance
 * Heartbeat survey to determine user satisfaction


Who is the owner of the data analysis for this study?
Saptarshi Guha (:joy)


Will this experiment require uplift?
No


QA Status of your code
PI request submitted. Results report will be available early in the week of Oct 30.


Do you plan on surveying users at the end of the study?
Yes, we want to get a general satisfaction score (5 stars help understand/compare the satisfaction level) followed-up by custom questions to help understand what was good or bad (i.e help understand if users were confused or happy or scared by the indicator)


Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:
 * Bug 1383051 - Add a visual indicator when accessibility is enabled
 * Bug 1384567 - Add a privacy preference for accessibility
 * https://support.mozilla.org/en-US/kb/accessibility-services
 * https://sql.telemetry.mozilla.org/queries/14971 
 * https://docs.google.com/document/d/1i9gcICi2TSHguuVcw9S800R3qER5EPbrRMXDcovv_1E/edit?ts=59ee32bf 
 * https://docs.google.com/spreadsheets/d/1P4EEjb9Yr6jAEFyeMLPOTjgu6E1w1CpE9hCWYjK9CiI/edit#gid=1165544449
 * https://docs.google.com/document/d/1VOPe_Yu2Wx5zB4TpMX-8GU3dynQlTFR25kfQRDhopyQ/edit

Comment 2

[Yura Zenevich [:yzen] (please use "needinfo")]

FYI, PI request for the features complete. No issues raised.

Comment 3

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking this bug for the 57 release to make sure everything is in place.

Comment 4

[David Bolter [:davidb] (NeedInfo me for attention)]

Adding some color. We want to ship this soon to mitigate a lot of performance issues that are cropping up on 57 release.

In particular Firefox accessibility instantiated via 'unknown' (and 'uiautomation') default to e10s (because single process is now officially unsupported) and so that combination is increasing. We expected unknown unknowns, and one appears to be a lot of users experiencing performance issues.

https://sql.telemetry.mozilla.org/dashboard/parent-a11y-consumers

This is approximately 7% of users
https://mzl.la/2zOShhS

If we can ship this indicator, a lot of users can discover and opt out of accessibility.
https://support.mozilla.org/en-US/kb/accessibility-services#w_should-i-disable-accessibility-services

Comment 5

[Yura Zenevich [:yzen] (please use "needinfo")]

Hi Matt, if I understand correctly, we need an explicit approval here before the intent to ship email? Thanks

Comment 6

[Matt Grimes [:Matt_G]]

This all looks good to me. I've reviewed the PHD and all the stakeholders are in agreement. I think you're ready to send the intent to ship email.

Comment 7

[Yura Zenevich [:yzen] (please use "needinfo")]

Attachment: Accessibility Icon 24x24.png

Comment 8

[Yura Zenevich [:yzen] (please use "needinfo")]

Attachment: Accessibility Icon 48x48.png

Comment 9

[Yura Zenevich [:yzen] (please use "needinfo")]

Joni updated the article including icons. Closing as Fixed.

Comment 10

[Yura Zenevich [:yzen] (please use "needinfo")]

Ignore comment 9, it belongs to the blocking bug.

Comment 11

[Matt Grimes [:Matt_G]]

We've worked with Yura and crew to identify a more appropriate population size. We'll be looking at:

* Release 58
* All locales
* 10% of the 7% of A11y population split 50-50 


Looking for relman to re-approve this since it was originally intended to ship in 57.

Comment 12

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

My understanding is that this study has been run in the past and QA's as well. Please go ahead and deploy this study.

Comment 13

[krupa raj [:krupa--use this to needinfo]]

Looks like this shield study was tested by Alexandru Simonca. Ship it.

Comment 14

[Robert (rrayborn, he/him)]

We're live.

Comment 15

[Josh Gaunt [:jgaunt]]

Taken down today.

Comment 16

[Yura Zenevich [:yzen] (please use "needinfo")]

The data suggests with enough statistical significance, that landing indicator as is can negatively affect user retention. The study also indicates that with users who clicked on the indicator retention improves overall user retention.

Outcomes:
This version of the indicator + support page should not be landed.

A follow up study is needed to test several other improvements to notifying the user via the indicator. After talking with Amie(UX), the list of approaches that we should try is:

* Original control group with no indicator.
* Original test group with just the indicator.
* New test group with the indicator that has different visuals.
* New test group with an indicator and a doorhanger that would have a very short blurb about performance and further actions.
* New test group with that will receive a system notification that would have a very short blurb about performance and further actions.

NI'ing Amy and Michelle regarding these options.

Comment 17

[Amy Lee [:amylee] UX]

Hi, 

These all seem reasonable to me. Since the study didn't mention any issues with visual appearance of the icon itself (i.e suspicious looking, etc.) I think we can omit testing new visuals. It seems the greatest impact would be actually testing putting the information upfront to users vs. requiring user action (i.e click on icon).

Comment 18

[Antony Hilliard]

Just chiming in as a casual user here - while diagnosing a Firefox crash related to accessibility ( https://bugzilla.mozilla.org/show_bug.cgi?id=1419108 ), I was surprised to find what seemed like malware: about:support showed that accessibility was Activated with Instantiator UNKNOWN| .  Not very trust inspiring.  I was even more surprised that the Accessibility visual indicator (which would have alerted me to the seeming malware) is disabled by default.  I hope you can agree on enabling this again soon, as a privacy & security issue.

Comment 19

[Dão Gottwald [:dao]]

What's the current status here? Are there still further steps planned?

Comment 20

[Yura Zenevich [:yzen] (please use "needinfo")]

This definitely needs another shield study if we consider enabling it by default. Doing so might be useful for products like Focus for desktop.