1528310 - [Shield] Pref-Flip Study: Windows launcher process, 67, Release

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Experimenter Bot]

Windows launcher process

The Windows launcher process intends to make the dll blocklist reliably work on startup, by first initiating a bootstrap process that manages the blocking before the main thread is started. We want to test that this new process doesn't have adverse effects on retention or user engagement.

More information: https://experimenter.services.mozilla.com/experiments/windows-launcher-process/

Comment 1

[Romain Testard [:RT]]

[Tracking Requested - why for this release]:

Comment 2

[Jess Mokrzecki [:jmok]]

Attachment: Requesting Design Review

Comment 3

[Pascal Chevrel:pascalc (PTO until April 26)]

Romain, can you give a status update on this experiment in the context of 67 (+trailhead)? Thanks

Comment 4

[Romain Testard [:RT]]

We still plan to start user enrolment May 21st. Since we had planned user enrolment over 2 weeks we are OK with starting the experiment as 67 ships (i.e throttling 67 won't prevent us from acquiring the users we need).

Comment 5

[Andreea Cupsa [:acupsa], Experiments QA]

Windows launcher process pref flip
Targeted: Firefox Release 67.0

We have finished testing the Windows launcher process pref flip experiment.
Please note that due to the fact that the “browser.launcherProcess.enabled” preference is set as a user preference, the Normandy recipe will have to be set accordingly and not target a default preference.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

• No new issues were uncovered during testing.

Testing summary:

• Full Functional test suite: TestRail

Tested Platforms:

• Windows 10 x64 with Avast

Tested Firefox versions:

• Firefox Release Candidate 67 build1 (locales: en-US, de, fr)
• Firefox Release Candidate 67 build2 (locales: en-US, de, fr)

Comment 6

[Experimenter Bot]

    Experiment Type: Pref Flip Study

    What is the preference we will be changing

browser.launcherProcess.enabled

    What are the branches of the study and what values should
    each branch be set to?

- Treatment Windows launcher process enabled 50%:

Value: true

The test branch will use the new Windows launcher process to start Firefox after the next restart.
        
- Treatment Windows launcher process not enabled 50%:

Value: false

Firefox will be started the ordinary way.
        

    What version and channel do you intend to ship to?

0.5% of Release Firefox 67.0

    Are there specific criteria for participants?

Locales:

Geographic regions:

Prefs:

Studies:

Any additional filters:
Windows

    What is your intended go live date and how long will the study run?

May 21, 2019 - Jun 25, 2019 (35 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

We are hoping not to see regressions in retention and markers of engagement:

- Total URI count
- Usage hours 
- 2 week retention

220,000 users in each branch is enough to detect a
- 3% regression in total URIs
- 2% regression in active hours, and
- .75% regression in 2 week retention
at 90% power with alpha=.01. For 2 branches this will be 440,000 users, or approximately .5% of the weekly Windows release population.

Do you plan on surveying users at the end of the study?
No.

    Who is the owner of the data analysis for this study?

wbeard

    Will this experiment require uplift?

False

    QA Status of your code:



    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/windows-launcher-process/