1481305 - [Shield] Pref Flip Study: Default Browser Study, release 61

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Tyler Downer [He/Him]]

Basic description of experiment: Promoting non-default browser users to set Firefox as default

What is the preference we will be changing? browser.shell.checkDefaultBrowser

What independent variable(s) (IVs) are you manipulating to affect measurements of your DV(s)? What different levels (values) can each IV assume? 
Prompt: browser.shell.checkDefaultBrowser TRUE
Control: Leave browser.shell.checkDefaultBrowser at False

What percentage of users do you want in each branch? 
50/50 each, 1% overall

What Channels and locales do you intend to ship to? 
Firefox 61.0.2 Release, en-US,

What is your intended go live date and how long will the study run? 
Aug 20th, 2018
Run for two weeks, monitoring overall usage

Are there specific criteria for participants? 
Non-Default, en-US, Windows, Mac

What is the main effect you are looking for and what data will you use to make these decisions? If a user accepts the prompt to set Firefox as default, does retention or any of our usage metrics go up? 

Who is the owner of the data analysis for this study? Ilana

Will this experiment require uplift? No, the reqired fix is already in 61.0.2

QA Status of your code: N/a

Do you plan on surveying users at the end of the study? No

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:
https://www.mozdatacollective.com/extcontent/index/index/element/1268/ Survey run on users asking their perceived defaultness

Comment 1

[MarniePW [:marnie]]

PHD: https://docs.google.com/document/d/1ERD4IqkKUwtFyelQ9q9IrZvNO4bfhTng1c07oGuWXJQ/edit

Comment 2

[MarniePW [:marnie]]

Ilana, any concerns with this one?

Comment 3

[MarniePW [:marnie]]

Remus, can you add QA signoff here, too?

Comment 4

[MarniePW [:marnie]]

Dave, can you get a peer to review this?

Comment 5

[MarniePW [:marnie]]

And Liz, FYI about this one.

Comment 6

[Ilana]

Science: R+

Comment 7

[Dave Townsend [:mossop]]

peer r+ however it is worth noting that an enterprise policy exists that can stop us from using the value of this preference. It's unlikely that many of our users have the policy applied but I'm not qualified to judge whether that will affect your numbers.

Comment 8

[Ilana]

Mossop, how will we know if users have this policy?

Comment 9

[Dave Townsend [:mossop]]

Felipe can probably help understand that. Felipe, can we tell from telemetry if a user has a specific policy applied?

Comment 10

[:Felipe Gomes (needinfo for replies!)]

We don't collect telemetry on policies at the moment, but I wouldn't worry about this policy for this study. Most users of policies are on the ESR branch, so as long as this doesn't target ESR 60.*, I think we're fine.

Does the pref flip system supports any feedback about the previous value of the pref, or whether it was locked or not? Checking if the pref is locked and excluding these users would be an alternative way to detect that, if it's possible to do so.

Comment 11

[Liz Henry (:lizzard)]

Ryan, passing this on to you for 61. 
I agree we aren't likely to have many enterprise policy users on 61 release.

Comment 12

[Remus Dranca [rdranca], Ecosystem QA]

We have finished testing the Default Browser Shield Study experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

- No issues have been found during testing.

Testing Summary:
- Full Functional test suite: TestRail.

Tested Platforms:

- Windows 10 x64

- Windows 7 x64

- Mac 10.13.3

Tested Firefox version:

- Firefox Release  61.0.2, en-US

- Firefox Beta 62.0b17, en-US


Comments:

- We have also tested the study on Firefox Beta 62.0b17, en-US, because of a comment that appeared in the PHD document and which stated that the run time of the study may need to be longer. We haven’t received a confirmation on this matter from developers due to PTO. 

Our concern was that the study could end up being live on both Firefox Release 61 and 62 with no coverage on the Beta channel.

Regards,
Remus

Comment 13

[Ryan VanderMeulen [:RyanVM]]

Looks good to me. Signing off for RelMan.

Comment 14

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

(Set flags for all approvals mentioned in the bug).

- Published:  https://delivery-console.prod.mozaws.net/recipe/555/
- Observed in field.:  New experiment id observed: Default_Browser_Pref-Flip: {"type":"normandy-exp","branch":"Control"}.

Comment 15

[Tyler Downer [He/Him]]

Disabled on Friday. Josh Gaunt is on the hook for analysis

Comment 16

[Tyler Downer [He/Him]]

Fixed. Pursuing full roll-out in Bug 1507011