[Experiment] Pref-Flip Experiment: Studying OS reauthentication success rate with improved user instructions in about:logins Fx 77.0 to 77.0 Release
Categories
(Shield :: Shield Study, task, P3)
Tracking
(firefox77+ fixed)
People
(Reporter: experimenter, Unassigned)
References
()
Details
User Story
Delivery Type: Pref Flip Experiment What is the preference we will be changing signon.management.page.os-auth.enabled What are the branches of the experiment and what values should each branch be set to? - Control Updated strings with improved help text 100%: Value: true Updated strings with improved help text What version and channel do you intend to ship to? 0.05% of Release Firefox 77.0 to 77.0 Are there specific criteria for participants? Prefs: Experiments: Any additional filters: Countries: all Locales: English (US) (en-US) What is your intended go live date and how long will the experiment run? Jun 09, 2020 - Jul 07, 2020 (28 days) What is the main effect you are looking for and what data will you use to make these decisions? What is the main effect you are looking for and what data will you use to make these decisions? What metrics are you using to measure success We are looking to see the pwmgr.reauthenticate Event telemetry where object=os_auth and value begins with "success" occur at least once for over 80% of users. We are currently seeing around 80% for this metric on Windows, and we hope to see this number increase though we aren't sure yet what that number should look like. Do you plan on surveying users at the end of the delivery? No. Who is the owner of the data analysis for this experiment? ksiegler@mozilla.com Will this experiment require uplift? False QA Status of your code: Finished testing the experiment and sent a YELLOW sign off. Link to more information about this experiment: https://experimenter.services.mozilla.com/experiments/studying-os-reauthentication-success-rate-with-improved-user-instructions-in-aboutlogins/
Studying OS reauthentication success rate with improved user instructions in about:logins
Bug 1194529 added a feature that will prompt the user for OS authentication before providing access to stored passwords. Telemetry showed a high number of users unable to successfully authenticate. This feature was disabled for all users on Firefox 76 and Firefox 77 in bug 1636511. In bug 1636032 (and a couple other follow-up bugs) we updated the instructions for the user as to which account password to use for authentication. These changes were landed in Firefox 78. In bug 1638908 I have a patch for uplift to Firefox 77 that contains the strings introduced by bug 1636032 et al.
After the changes from bug 1638908 are uplifted to beta, we would like to have a small rollout tested with users on Windows to see if their success rate improves. A success is measured by a Firefox profile that has logged at least one successful authentication with the OS reauthentication feature.
Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.
Data Science Issue: https://jira.mozilla.com/browse/DO-235
More information: https://experimenter.services.mozilla.com/experiments/studying-os-reauthentication-success-rate-with-improved-user-instructions-in-aboutlogins/
Comment 1•4 years ago
|
||
Studying OS reauthentication success rate with improved user instructions in about:logins
Firefox Release 77.x
We have finished testing the “[PI-647] Studying OS reauthentication success rate with improved user instructions in about:logins” experiment.
Quality status: YELLOW - Ship It Conditionally
Why is this experiment YELLOW?
- During testing, we have found one new issue: Bug 1643654.
- There are 3 known issues that might affect the user's experience.
- Due to the time constraints, we have tested the experiment only on Windows 10 x64 1909, Windows 10 64x 1903, Windows 8.1 x32 9600 and Windows 7 x64 versions. Considering this we cannot be sure that there are no other failures on other Windows versions.
- None of the issues are considered as blocker for the experiment from QA point of view. However, we are unsure if the new Telemetry issue found would affect or not the purpose of the study.
Testing Summary:
- Test Suite: https://jira.mozilla.com/browse/PI-647
Tested Platforms:
- Windows 10 x64 1909;
- Windows 10 x64 1903;
- Windows 8.1 x32 9600;
- Windows 7 x64;
Tested Firefox versions:
- Firefox 77.0 en-US build;
- Firefox 77.0.1 en-US build;
Comment 2•4 years ago
|
||
[Tracking Requested - why for this release]: Experimenter study for Firefox 77 release
Updated•4 years ago
|
Reporter | ||
Updated•4 years ago
|
Reporter | ||
Comment 3•4 years ago
|
||
Start Date: 2020-06-09 End Date: 2020-07-07
Reporter | ||
Updated•4 years ago
|
Updated•4 years ago
|
Description
•