1400942 - [Shield] Pref Flip Study: Session Restore Button

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Peter Dolanjski [:pdol]]

Basic description of experiment: 
Session restore (ability to restore all previously open tabs) is currently invoked manually and is also not discoverable.  The proposal in Bug 1219725 is to add a button to the chrome of Firefox to allow users to invoke session restore in a discoverable way.  Before shipping the functionality to a release population, we want to ensure the implementation would actually be used by users and result in a higher number of restored sessions. 

What is the preference we will be changing? 
browser.tabs.restorebutton

What are the branches of the study and what values should each branch be set to? 
Control: browser.tabs.restorebutton set to 2
Variant: browser.tabs.restorebutton set to 1

What percentage of users do you want in each branch? 
50% in each

What Channels and locales do you intend to ship to? 
Beta 56
Release 56 (when it ships)
All locales

What is your intended go live date and how long will the study run? 
Go live for beta: September 21th
Ending: October 5th
Go live for release: September 28th
Ending: October 12th

Are there specific criteria for participants? 
No

What is the main effect you are looking for and what data will you use to make these decisions? 
-How often does the button get shown? (totals, average display per user, percentage of users that see it, in a given time period)
-At what rate is the button actually used? (as a percentage of how many times it is shown and also as a percentage of unique users that it is shown to in a given time period.) 
--We are collecting a baseline, so are not adding success criteria (our hypothesis is that a 10% or greater rate (used as a % of times it is shown and as % of unique users that make use of the feature relative to how many saw the button) would be considered successful)
-How many sessions are restored via this button vs. other ways? (as a percentage)
--We are collecting a baseline, so are not adding success criteria (our hypothesis is that a 40% or greater share of restores using this method would be considered successful)
-How many tabs are restored via the button? (average, median, 90th percentile)
-Does the button have an impact on total uris loaded in experiment group relative to control?
--A statistically significant increase of at least 2% would be considered a success

Who is the owner of the data analysis for this study? 
Ilana Segall?

Will this experiment require uplift? 
No

QA Status of your code: 
Green (results: https://public.etherpad-mozilla.org/p/Session_Restore_Button_Pref_Flip_Experiment_(56))

Do you plan on surveying users at the end of the study? 
Yes

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:
N/A

Comment 1

[Matt Grimes [:Matt_G]]

What's the urgency for shipping in Beta this week? I'd suggest waiting until 56 hits release and turning on both Beta and Release.

Comment 2

[Matt Grimes [:Matt_G]]

Hey pdol. Do you want this live in Beta 57 only or do you want to jump to Release 56. Let me know.

Comment 3

[Peter Dolanjski [:pdol]]

(In reply to Matt Grimes [:Matt_G] from comment #2)
> Hey pdol. Do you want this live in Beta 57 only or do you want to jump to
> Release 56. Let me know.

Release only due to timelines

Comment 4

[Josh Gaunt [:jgaunt]]

recipe enabled for 1% sample

Comment 6

[Matt Grimes [:Matt_G]]

@ilana @jgaunt Now that this study has ended can you please recap the outcome here and close the bug? A couple sentences is fine.

Comment 8

[Dão Gottwald [:dao]]

(In reply to Matt Grimes [:Matt_G] from comment #6)
> @ilana @jgaunt Now that this study has ended can you please recap the
> outcome here and close the bug? A couple sentences is fine.

Peter?

Comment 9

[:shell escalante]

closing due to age and this study not running currently.