Closed Bug 1464545 Opened 7 years ago Closed 7 years ago

[Shield] Pref Flip Study: Windows Restart Manager in release 62

Categories

(Shield :: Shield Study, enhancement, P1)

Product:
Shield
Component:
Shield Study
Platform:
Unspecified
Windows
Type:
enhancement

Tracking

(firefox60 unaffected, firefox61 unaffected, firefox62+ wontfix, firefox63 unaffected)

Status:
RESOLVED WONTFIX
Tracking Flags:
Tracking Status
firefox60 --- unaffected
firefox61 --- unaffected
firefox62 + wontfix
firefox63 --- unaffected

People

(Reporter: cpeterson, Assigned: cpeterson)

References

Details

User Story

# Basic description of experiment

The Windows Restart Manager will automatically restart Firefox after a Windows OS reboot (manual user shutdown or automatic Windows Update reboot). We expect users will have higher engagement (because their previous Firefox session is restored) and no drop in retention.

# What is the preference we will be changing

toolkit.winRegisterApplicationRestart

# What are the branches of the study and what values should each branch be set to?

* Test branch: pref = true. Firefox will restart automatically after reboot.
* Control branch: pref = false. Firefox will not restart automatically. This is the current behavior.

# What percentage of users do you want in each branch

We need a sample size of 376,000 users in the test branch. 50/50% split between test and control branches.

# What Channels and locales do you intend to ship to?

Channel: Firefox 62 Release
Locale: en-US only, but this needs to be confirmed!

# What is your intended go live date and how long will the study run?

* Enrollment will run for one week starting September 10.
* The experiment will then run for three weeks starting September 

# Are there specific criteria for participants?

Yes:

* Windows only
* Existing users only
* Test cohort targeted on telemetry.os_shutting_down == true in normandy (to limit to the 4.7% win users that shutdown with Fx enabled)

# What is the main effect you are looking for and what data will you use to make these decisions?

We expect users in the test branch will have higher engagement and no drop in retention. We would like to monitor the following probes:

* Unique URI count. We will measure unique URIs to avoid double counting of URIs in restored tabs.
* 3 week retention

# Who is the owner of the data analysis for this study?

Saptarshi

# Will this experiment require uplift?

No

# QA Status of your code:

QA's mid-Nightly status is currently yellow because we have some open bugs, but these bugs will be fixed long before 62 rides to the Release channel in September.

# Do you plan on surveying users at the end of the study?

Yes. Here is a link to the survey request (WIP):

https://docs.google.com/document/d/1Vx3mFGulsIAxlVeYZMt2pm033GFHnsU9whyTNb_Gl68/

# Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:

* Study's PHD: https://docs.google.com/document/d/1uMCznVZXnHAHzbguLkl7UApQweVyel55e2u8HK9lJXs/
* Study's Trello card: https://trello.com/c/jUob6uQJ/
No description provided.
Ilana, here is the bug for the Windows Restart Manager Shield study.
status-firefox60: wontfix → unaffected
status-firefox61: wontfix → unaffected
Flags: needinfo?(isegall)
Science review: R+
Flags: needinfo?(isegall)
Blocks: 603903
No longer depends on: 603903
Science Review: R+
Is there anything currently planned here for 62?
tracking-firefox62: --- → +
Flags: needinfo?(cpeterson)
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #4) > Is there anything currently planned here for 62? Yes. We plan to run a Shield study in 62 release from September 10 - October 8, followed by a user survey. Is there any particular information you need now for 62's release management? Trello card tracking our Shield study: https://trello.com/c/jUob6uQJ/96-register-with-windows-restart-manager
Assignee: nobody → cpeterson
status-firefox63: --- → unaffected
Flags: needinfo?(cpeterson)
Blocks: 1480144
Priority: -- → P1
WONTFIX because we no longer plan to run a Shield study. We are shipping the feature in 61.0.2 and using a Normandy rollout instead of a Shield study. Saptarshi will still be helping to analyze the telemetry probes outlined in comment 0 during the rollout.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WORKSFORME
Reopening. There was a problem with the telemetry analysis of 61.0.2's pref rollout so we need to run this pref flip study in Firefox 62 after all.
Status: RESOLVED → REOPENED
status-firefox62: wontfix → affected
Resolution: WORKSFORME → ---
This one's coming in hot, Ryan...
Flags: needinfo?(ryanvm)
Summary: [Shield] Pref Flip Study: Windows Restart Manager in 62 → [Shield] Pref Flip Study: Windows Restart Manager in release 62
Per today's discussion, I believe this is back into the land of WONTFIX.
Status: REOPENED → RESOLVED
Closed: 7 years ago → 7 years ago
status-firefox62: affected → wontfix
Flags: needinfo?(ryanvm)
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.