1400980 - [Shield] Pref Flip Study: Tracking Protection UI Pref

Status: RESOLVED WONTFIX

(Shield :: Shield Study, defect)

Description

[Peter Dolanjski [:pdol]]

Basic description of experiment: 
We will be using a Shield Pref Flip study to do an incremental roll out of the Tracking Protection UI Pref before full roll out.
By setting the pref below to true, it will enable the UI Pref that exposes the setting in Preferences that allows the user to enable Tracking Protection outside of Private Browsing if they so choose.
We are confident that the breakage of websites is low enough (see study below) to justify exposing this preference to users.  However, by rolling out incrementally, we can monitor uptake of the use of this feature and decide whether or not to hold off further deployment.

What is the preference we will be changing? 
privacy.trackingprotection.ui.enabled

What are the branches of the study and what values should each branch be set to? 
Control: privacy.trackingprotection.ui.enabled set to false
Variant: privacy.trackingprotection.ui.enabled set to true

What percentage of users do you want in each branch? 
80% in variant, 20% control, up to approximately 10% of release users.

What Channels and locales do you intend to ship to? 
Release 56 (continuing into release 57)

What is your intended go live date and how long will the study run? 
Go live: October 9th
Ending: November 9th

Are there specific criteria for participants? 
No

What is the main effect you are looking for and what data will you use to make these decisions? 
- We are looking to see if the % of users in the treatment group who enable tracking protection (Always) is high relative to the control group in a 3 week period.
Full deployment will be based on uptake in the period (for the experiment group) being no more than 10% higher relative to those in control.

Who is the owner of the data analysis for this study? 
Francois Marier

Will this experiment require uplift? 
No

QA Status of your code: 
Green - been in Nightly for years

Do you plan on surveying users at the end of the study? 
No

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:
Privacy Prefs Breakage Study: https://docs.google.com/presentation/d/1OVtXAnyeBLX2N1yyZoTMP9AV_6HnI3mnXwIFlOL7yOA/edit?ts=59a888d2#slide=id.g251dbe7f10_0_14

Comment 1

[Peter Dolanjski [:pdol]]

Matt, please review before I sent the intent to ship.

Comment 2

[Peter Dolanjski [:pdol]]

Note: we'll review the uptake rates during this study cycle, but the 3 week period will start upon the release of v57.

Comment 3

[Matt Grimes [:Matt_G]]

This one looks pretty straight forward. Sorry for the delay. r+

Comment 4

[Josh Gaunt [:jgaunt]]

Approved for 10% sample on beta

Comment 5

[Peter Dolanjski [:pdol]]

(In reply to Peter Dolanjski [:pdol] from comment #0)
> What is the main effect you are looking for and what data will you use to
> make these decisions? 
> - We are looking to see if the % of users in the treatment group who enable
> tracking protection (Always) is high relative to the control group in a 3
> week period.
> Full deployment will be based on uptake in the period (for the experiment
> group) being no more than 10% higher relative to those in control.

Note: I am revising this threshold to 30% since the % of session that use TP now is so low (1.5%).

Comment 6

[François Marier [:francois]]

This experiment is no longer needed: https://bugzilla.mozilla.org/show_bug.cgi?id=1393627#c5