Closed Bug 1636591 Opened 4 years ago Closed 4 years ago

[Experiment] Pref-Flip Experiment: DevTools Experiment: Prevent accidental use of F12 Fx 78.0 to 78.0 Beta

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: experimenter, Assigned: Harald)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

devtools.experiment.f12.shortcut_disabled

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment Control 50%:

Value: false

No changes
        
- Control Test 50%:

Value: true

F12 disabled for non-DevTools users
        

    What version and channel do you intend to ship to?

100% of Beta Firefox 78.0 to 78.0

    Are there specific criteria for participants?

Prefs:

Experiments:

Any additional filters:
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Jun 02, 2020 - Jul 02, 2020 (30 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

Users of Beta accidentally hit F12 which opens the DevTools . This can often confuse them (and they leave it open). or they simply close it. This is a bad user experience especially since they had no interest in opening DevTools in the first place and also. it confounds our retention metrics (suppresses the true value).

In this experiment,  for the test group F12 is blocked. In other words, if the user presses it, they are asked to open  Devtools from the menu which will open  Devtools. The control group receive the existing experience - F12 opens DEvtools.

What do we expect to see?

1. Between test and control, the click rate of F12 ought be the same

2.a We want to measure what % of the F12 clickers (who click for the first time) actually go on to open DevTools
2.b Among this group (who opened Devtools) we expect their retention to be higher than the Control group (proportion who open Devtools again in subsequent 7 days)

3. Among users who click F12, and don't open Devtools, do they continue (to accidentally?)  click on F12?

4. We cant really detect accidental users with certainty from the control. yes, some users will open and close quickly (say within 10 seconds), but often others will leave it open(anecdata from Harald). One thing we can explore is to  compare accidental users (see mbalfanz's query (https://sql.telemetry.mozilla.org/queries/68917/source#174005) and use the definition of accidental users from control and compare this the group in test who dont click on Devtool menu (having clicked on F12).

We might break out as a separate group, users who have used Devtools before , based on `devtools.selfxss.count>=5`.

(Sanity check, the 1-proportion(2a) ought be roughly similar to proportion of accidental users in control)

    Who is the owner of the data analysis for this experiment?

sguha@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

Not started

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/devtools-experiment-prevent-accidental-use-of-f12/

DevTools Experiment: Prevent accidental use of F12

We want to disable F12 for non-DevTools users to avoid accidental openings of DevTools. This should lead to better retention/engagement while improving the quality of our metrics of DevTools usage.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DS-586
More information: https://experimenter.services.mozilla.com/experiments/devtools-experiment-prevent-accidental-use-of-f12/

DevTools Experiment: Prevent accidental use of F12
Firefox Beta 78

We have finished testing the PI-608 - DevTools Experiment: Prevent accidental use of F12 experiment.

Quality status: GREEN - SHIP IT

Why is this experiment green?

  • We have found 3 issues while testing the experiment. All the issues have low severity and do not impact the end users in a harmful way.

Testing Summary:

Tested Platforms:

  • Windows 10 x64
  • Linux Mint 19.2 Tina
  • macOS 10.15

Tested browser versions:

  • Firefox Nightly 78.0a1 (en-US, en-CA, en-GB)
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: DevTools Experiment: Prevent accidental use of F12 → [Experiment] Pref-Flip Experiment: DevTools Experiment: Prevent accidental use of F12 Fx 78.0 to 78.0 Beta
Start Date: 2020-06-09 End Date: 2020-07-09
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.