Closed Bug 1642402 Opened 4 years ago Closed 4 years ago

[Experiment] Pref-Flip Experiment: Studying OS reauthentication success rate with improved user instructions in about:logins Fx 77.0 to 77.0 Release

Categories

(Shield :: Shield Study, task, P3)

Tracking

(firefox77+ fixed)

RESOLVED FIXED
Tracking Status
firefox77 + fixed

People

(Reporter: experimenter, Unassigned)

References

()

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

signon.management.page.os-auth.enabled

    What are the branches of the experiment and what values should
    each branch be set to?

- Control Updated strings with improved help text 100%:

Value: true

Updated strings with improved help text
        

    What version and channel do you intend to ship to?

0.05% of Release Firefox 77.0 to 77.0

    Are there specific criteria for participants?

Prefs:

Experiments:

Any additional filters:
Countries: all

Locales: English (US) (en-US) 

    What is your intended go live date and how long will the experiment run?

Jun 09, 2020 - Jul 07, 2020 (28 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

We are looking to see the pwmgr.reauthenticate Event telemetry where object=os_auth and value begins with "success" occur at least once for over 80% of users. We are currently seeing around 80% for this metric on Windows, and we hope to see this number increase though we aren't sure yet what that number should look like.

Do you plan on surveying users at the end of the delivery? No.

    Who is the owner of the data analysis for this experiment?

ksiegler@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

Finished testing the experiment and sent a YELLOW sign off.

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/studying-os-reauthentication-success-rate-with-improved-user-instructions-in-aboutlogins/

Studying OS reauthentication success rate with improved user instructions in about:logins

Bug 1194529 added a feature that will prompt the user for OS authentication before providing access to stored passwords. Telemetry showed a high number of users unable to successfully authenticate. This feature was disabled for all users on Firefox 76 and Firefox 77 in bug 1636511. In bug 1636032 (and a couple other follow-up bugs) we updated the instructions for the user as to which account password to use for authentication. These changes were landed in Firefox 78. In bug 1638908 I have a patch for uplift to Firefox 77 that contains the strings introduced by bug 1636032 et al.

After the changes from bug 1638908 are uplifted to beta, we would like to have a small rollout tested with users on Windows to see if their success rate improves. A success is measured by a Firefox profile that has logged at least one successful authentication with the OS reauthentication feature.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-235
More information: https://experimenter.services.mozilla.com/experiments/studying-os-reauthentication-success-rate-with-improved-user-instructions-in-aboutlogins/

Studying OS reauthentication success rate with improved user instructions in about:logins
Firefox Release 77.x

We have finished testing the “[PI-647] Studying OS reauthentication success rate with improved user instructions in about:logins” experiment.

Quality status: YELLOW - Ship It Conditionally

Why is this experiment YELLOW?

  • During testing, we have found one new issue: Bug 1643654.
  • There are 3 known issues that might affect the user's experience.
  • Due to the time constraints, we have tested the experiment only on Windows 10 x64 1909, Windows 10 64x 1903, Windows 8.1 x32 9600 and Windows 7 x64 versions. Considering this we cannot be sure that there are no other failures on other Windows versions.
  • None of the issues are considered as blocker for the experiment from QA point of view. However, we are unsure if the new Telemetry issue found would affect or not the purpose of the study.

Testing Summary:

Tested Platforms:

  • Windows 10 x64 1909;
  • Windows 10 x64 1903;
  • Windows 8.1 x32 9600;
  • Windows 7 x64;

Tested Firefox versions:

  • Firefox 77.0 en-US build;
  • Firefox 77.0.1 en-US build;

[Tracking Requested - why for this release]: Experimenter study for Firefox 77 release

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Studying OS reauthentication success rate with improved user instructions in about:logins → [Experiment] Pref-Flip Experiment: Studying OS reauthentication success rate with improved user instructions in about:logins Fx 77.0 to 77.0 Release
Start Date: 2020-06-09 End Date: 2020-07-07
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.