[Experiment] Pref-Flip Experiment: Studying OS reauthentication success rate with improved user instructions in about:logins V2 Fx 77.0 to 79.0 Release
Categories
(Shield :: Shield Study, task, P3)
Tracking
(Not tracked)
People
(Reporter: experimenter, Assigned: ksiegler)
References
()
Details
User Story
Delivery Type: Pref Flip Experiment
What is the preference we will be changing
signon.management.page.os-auth.enabled
What are the branches of the experiment and what values should
each branch be set to?
- Control Updated strings with improved help text 100%:
Value: true
Updated strings with improved help text
What version and channel do you intend to ship to?
0.6% of Release Firefox 77.0 to 79.0
Are there specific criteria for participants?
Prefs:
Experiments:
Any additional filters:
Countries: all
Locales: English (US) (en-US)
What is your intended go live date and how long will the experiment run?
Jul 21, 2020 - Aug 18, 2020 (28 days)
What is the main effect you are looking for and what data will you use to
make these decisions?
What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success
We are looking to see the pwmgr.reauthenticate Event telemetry where object=os_auth and value begins with "success" occur at least once for over 80% of users. We are currently seeing around 80% for this metric on Windows, and we hope to see this number increase though we aren't sure yet what that number should look like.
Do you plan on surveying users at the end of the delivery? No.
Who is the owner of the data analysis for this experiment?
ksiegler@mozilla.com
Will this experiment require uplift?
False
QA Status of your code:
Finished testing the experiment and sent a YELLOW sign off.
Link to more information about this experiment:
https://experimenter.services.mozilla.com/experiments/studying-os-reauthentication-success-rate-with-improved-user-instructions-in-aboutlogins-v2/
Studying OS reauthentication success rate with improved user instructions in about:logins V2
We are testing a feature that will prevent casual snooping of saved passwords in Firefox Lockwise (about:logins). This feature prompts and validates the user's operating system account credentials before allowing access to saved passwords. When enrolled in this study, a prompt will be shown when trying to copy or reveal a saved password, or trying to edit a saved login.
Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.
Data Science Issue: https://jira.mozilla.com/browse/DO-235
More information: https://experimenter.services.mozilla.com/experiments/studying-os-reauthentication-success-rate-with-improved-user-instructions-in-aboutlogins-v2/
|
||
Comment 1•5 years ago
|
||
Studying OS reauthentication success rate with improved user instructions in about:logins
Firefox Release 77, 78
We have finished testing the [[PI-705] Studying OS reauthentication success rate with improved user instructions in about:logins experiment.
QA’s recommendation: YELLOW - SHIP IT, CONDITIONALLY
Why is this experiment yellow?
- Although during retesting, we didn’t find any new issues, there are 4 known issues that might affect the user's experience.
- Due to the time constraints, we have tested the experiment only on Windows 10 x64 1909, Windows 8.1 x32 9600 and Windows 7 x64 versions. Considering this we cannot be sure that there are no other failures on other Windows versions.
- None of the issues are considered as blocker for the experiment from a QA point of view. However, we are unsure if the Telemetry issue found would affect or not the purpose of the study.
Testing Summary:
- Test Suite - Testrail
Tested Platforms:
- Windows 10 x64 1909;
- Windows 8.1 x32 9600;
- Windows 7 x64;
Tested Firefox versions:
- Firefox 77.0.1 en-US build;
- Firefox 78.0 en-US build;
- Firefox 78.0.1 en-US build;
Regards,
Simona Rosu
Ecosystem QA Team
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Comment 2•5 years ago
|
||
|
|
Reporter | |
Updated•5 years ago
|
Description
•