Closed Bug 1412603 Opened 4 years ago Closed 2 years ago

[Shield] Pref Flip Study: accessibility visual indicator and permission preference

Categories

(Shield :: Shield Study, defect)

defect
Not set
normal

Tracking

(firefox57+ wontfix)

RESOLVED FIXED
Tracking Status
firefox57 + wontfix

People

(Reporter: yzen, Assigned: yzen)

References

Details

Attachments

(2 files)

Basic description of experiment:
Create 6 different cohorts that have the same user characteristics (representative sample of the release population). Proposed cohorts in detail:
 * New users that are a11y users (A11Y_INSTANTIATED_FLAG telemetry) and have default preference (indicator disabled - default)
 * Existing users that are a11y users (A11Y_INSTANTIATED_FLAG telemetry) and have 
default preference (indicator disabled - default)
 * New user that are a11y users (A11Y_INSTANTIATED_FLAG telemetry) and have preference changed (indicator enabled)
 * Existing users that are a11y users (A11Y_INSTANTIATED_FLAG telemetry) and have preference changed (indicator enabled)
 * New users that are not a11y users (indicator disabled - default)
 * Existing users that are not a11y users (indicator disabled - default)


What is the preference we will be changing?
'accessibility.indicator.enabled'


What are the branches of the study and what values should each branch be set to?
(indicator enabled) groups above: true
(indicator disabled - default) groups above: false


What percentage of users do you want in each branch?
TBC with :joy


What Channels and locales do you intend to ship to?
Release channel, all locales, 57 target 


What is your intended go live date and how long will the study run?
TBC with :joy


Are there specific criteria for participants?
No, both new and existing users


What is the main effect you are looking for and what data will you use to make these decisions?
 * Session length
 * Subsession length
 * 2 week retention
 * jank / hangs / performance
 * Heartbeat survey to determine user satisfaction


Who is the owner of the data analysis for this study?
Saptarshi Guha (:joy)


Will this experiment require uplift?
No


QA Status of your code
PI request submitted. Results report will be available early in the week of Oct 30.


Do you plan on surveying users at the end of the study?
Yes, we want to get a general satisfaction score (5 stars help understand/compare the satisfaction level) followed-up by custom questions to help understand what was good or bad (i.e help understand if users were confused or happy or scared by the indicator)


Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:
 * Bug 1383051 - Add a visual indicator when accessibility is enabled
 * Bug 1384567 - Add a privacy preference for accessibility
 * https://support.mozilla.org/en-US/kb/accessibility-services
 * https://sql.telemetry.mozilla.org/queries/14971 
 * https://docs.google.com/document/d/1i9gcICi2TSHguuVcw9S800R3qER5EPbrRMXDcovv_1E/edit?ts=59ee32bf 
 * https://docs.google.com/spreadsheets/d/1P4EEjb9Yr6jAEFyeMLPOTjgu6E1w1CpE9hCWYjK9CiI/edit#gid=1165544449
 * https://docs.google.com/document/d/1VOPe_Yu2Wx5zB4TpMX-8GU3dynQlTFR25kfQRDhopyQ/edit
Duplicate of this bug: 1411590
Flags: needinfo?(glind)
FYI, PI request for the features complete. No issues raised.
Depends on: 1411591
No longer depends on: 1411590
Tracking this bug for the 57 release to make sure everything is in place.
Depends on: 1408113
Depends on: 1418371
Adding some color. We want to ship this soon to mitigate a lot of performance issues that are cropping up on 57 release.

In particular Firefox accessibility instantiated via 'unknown' (and 'uiautomation') default to e10s (because single process is now officially unsupported) and so that combination is increasing. We expected unknown unknowns, and one appears to be a lot of users experiencing performance issues.

https://sql.telemetry.mozilla.org/dashboard/parent-a11y-consumers

This is approximately 7% of users
https://mzl.la/2zOShhS

If we can ship this indicator, a lot of users can discover and opt out of accessibility.
https://support.mozilla.org/en-US/kb/accessibility-services#w_should-i-disable-accessibility-services
Hi Matt, if I understand correctly, we need an explicit approval here before the intent to ship email? Thanks
Flags: needinfo?(glind) → needinfo?(mgrimes)
This all looks good to me. I've reviewed the PHD and all the stakeholders are in agreement. I think you're ready to send the intent to ship email.
Flags: needinfo?(mgrimes)
Joni updated the article including icons. Closing as Fixed.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Ignore comment 9, it belongs to the blocking bug.
We've worked with Yura and crew to identify a more appropriate population size. We'll be looking at:

* Release 58
* All locales
* 10% of the 7% of A11y population split 50-50 


Looking for relman to re-approve this since it was originally intended to ship in 57.
My understanding is that this study has been run in the past and QA's as well. Please go ahead and deploy this study.
Looks like this shield study was tested by Alexandru Simonca. Ship it.
We're live.
Taken down today.
The data suggests with enough statistical significance, that landing indicator as is can negatively affect user retention. The study also indicates that with users who clicked on the indicator retention improves overall user retention.

Outcomes:
This version of the indicator + support page should not be landed.

A follow up study is needed to test several other improvements to notifying the user via the indicator. After talking with Amie(UX), the list of approaches that we should try is:

* Original control group with no indicator.
* Original test group with just the indicator.
* New test group with the indicator that has different visuals.
* New test group with an indicator and a doorhanger that would have a very short blurb about performance and further actions.
* New test group with that will receive a system notification that would have a very short blurb about performance and further actions.

NI'ing Amy and Michelle regarding these options.
Flags: needinfo?(mheubusch)
Flags: needinfo?(amlee)
Hi, 

These all seem reasonable to me. Since the study didn't mention any issues with visual appearance of the icon itself (i.e suspicious looking, etc.) I think we can omit testing new visuals. It seems the greatest impact would be actually testing putting the information upfront to users vs. requiring user action (i.e click on icon).
Flags: needinfo?(amlee)
Just chiming in as a casual user here - while diagnosing a Firefox crash related to accessibility ( https://bugzilla.mozilla.org/show_bug.cgi?id=1419108 ), I was surprised to find what seemed like malware: about:support showed that accessibility was Activated with Instantiator UNKNOWN| .  Not very trust inspiring.  I was even more surprised that the Accessibility visual indicator (which would have alerted me to the seeming malware) is disabled by default.  I hope you can agree on enabling this again soon, as a privacy & security issue.
What's the current status here? Are there still further steps planned?
Flags: needinfo?(yzenevich)
See Also: → 1426131
This definitely needs another shield study if we consider enabling it by default. Doing so might be useful for products like Focus for desktop.
Flags: needinfo?(yzenevich)
Status: REOPENED → RESOLVED
Closed: 3 years ago2 years ago
Resolution: --- → FIXED
Flags: needinfo?(mheubusch)
You need to log in before you can comment on or make changes to this bug.