1526061 - [Shield] Pref-Flip Study: WebRender Performance 67, Release

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Experimenter Bot]

WebRender Performance 67

We believe that enabling WebRender by default does not negatively affect crash statistics nor checkerboarding telemetry values.

More information: https://experimenter.services.mozilla.com/experiments/webrender-performance-67/

Comment 1

[Thomas Elin [:relaas]]

[Tracking Requested - why for this release]:

Comment 2

[Andreea Cupsa [:acupsa], Experiments QA]

WebRender Performance 67
Targeted: Firefox Release 67.0

We have finished testing the WebRender for 67 experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

• No new issues were uncovered during testing. However, we encountered 3 known issues, which are low impact and do not block the launch of the experiment.

Testing summary:

• Full Functional test suite: TestRail

Tested Platforms:

• Windows 10 x64 with Nvidia GT 1030
• Windows 10 x64 with Nvidia GT 730

Tested Firefox versions:

• Firefox Beta 67.0b15
• Firefox Beta 67.0b18
• Firefox Release Candidate 67 build1
• Firefox Release Candidate 67 build2

Comment 3

[Experimenter Bot]

    Experiment Type: Pref Flip Study

    What is the preference we will be changing

gfx.webrender.all.qualified

    What are the branches of the study and what values should
    each branch be set to?

- Treatment WebRender Default in Fx67 is Enabled 50%:

Value: true

In 67 Release the WebRender default setting will still be Enabled.  So our control group has it as Enabled.
        
- Treatment WebRender Test in Fx67 is a disabled state 50%:

Value: false

In 67 Release the WebRender default setting will still be Enabled.  So our branch/test group has it as Disabled.
        

    What version and channel do you intend to ship to?

5% of Release Firefox 67.0

    Are there specific criteria for participants?

Locales: All Locales

Geographic regions: ALL

Prefs: -

Studies: -

Any additional filters: normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == 'available'

There is a staged roll-out for webrender in 67 also. we would like to avoid overlap, with the rollout deploying to clients that are not enrolled in the experiment.https://bugzilla.mozilla.org/show_bug.cgi?id=1541488

    What is your intended go live date and how long will the study run?

May 21, 2019 - Jun 11, 2019 (21 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

overall crash reports ≤ 5% increase in crash rate
OOM crash reports ≤ 5% increase in crash rate
shutdown crashes ≤ 5% increase in crash rate
CANVAS_WEBGL_SUCCESS ≤ 5% regression in median of fraction "True" per user
COMPOSITE_TIME Median per-user fraction of slow frames < 0.5% (absolute)
CONTENT_FRAME_TIME_VSYNC ≤ 5% regression in median of per-user fraction of slow events
CONTENT_FULL_PAINT_TIME ≤ 5% regression in fraction of slow paints (> 16 ms), ≤ 5% regression in median of per-user means
DEVICE_RESET_REASON ≤ 5% increase in reset rate
FX_PAGE_LOAD_MS_2 ≤ 5% regression in median of per-user means
FX_TAB_SWITCH_COMPOSITE_E10S_MS ≤ 5% regression in median of per-user means

Crash (/submission/reset) rates will be measured as events per 1,000 usage hours. A "slow event" means 16 ms or 200% vsync.

    Who is the owner of the data analysis for this study?

Tim Smith

    Will this experiment require uplift?

False

    QA Status of your code:

Is this study a minor revision of a previous study?
YES! See; https://bugzilla.mozilla.org/show_bug.cgi?id=1492568
See previous PHD; https://docs.google.com/document/d/1mo76Ub0l5cNIII0oKqoVn25Dop4nLKKf0CVhvK4_Wps/edit

    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/webrender-performance-67/

Comment 4

[Experimenter Bot]

    Experiment Type: Pref Flip Study

    What is the preference we will be changing

gfx.webrender.all.qualified

    What are the branches of the study and what values should
    each branch be set to?

- Treatment WebRender Default in Fx67 is Enabled 50%:

Value: true

In 67 Release the WebRender default setting will still be Enabled.  So our control group has it as Enabled.
        
- Treatment WebRender Test in Fx67 is a disabled state 50%:

Value: false

In 67 Release the WebRender default setting will still be Enabled.  So our branch/test group has it as Disabled.
        

    What version and channel do you intend to ship to?

5% of Release Firefox 67.0

    Are there specific criteria for participants?

Locales: All Locales

Geographic regions: ALL

Prefs: -

Studies: -

Any additional filters: normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == 'available'

There is a staged roll-out for webrender in 67 also. we would like to avoid overlap, with the rollout deploying to clients that are not enrolled in the experiment.https://bugzilla.mozilla.org/show_bug.cgi?id=1541488

    What is your intended go live date and how long will the study run?

May 21, 2019 - Jun 11, 2019 (21 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

overall crash reports ≤ 5% increase in crash rate
OOM crash reports ≤ 5% increase in crash rate
shutdown crashes ≤ 5% increase in crash rate
CANVAS_WEBGL_SUCCESS ≤ 5% regression in median of fraction "True" per user
COMPOSITE_TIME Median per-user fraction of slow frames < 0.5% (absolute)
CONTENT_FRAME_TIME_VSYNC ≤ 5% regression in median of per-user fraction of slow events
CONTENT_FULL_PAINT_TIME ≤ 5% regression in fraction of slow paints (> 16 ms), ≤ 5% regression in median of per-user means
DEVICE_RESET_REASON ≤ 5% increase in reset rate
FX_PAGE_LOAD_MS_2 ≤ 5% regression in median of per-user means
FX_TAB_SWITCH_COMPOSITE_E10S_MS ≤ 5% regression in median of per-user means

Crash (/submission/reset) rates will be measured as events per 1,000 usage hours. A "slow event" means 16 ms or 200% vsync.

    Who is the owner of the data analysis for this study?

Tim Smith

    Will this experiment require uplift?

False

    QA Status of your code:

Is this study a minor revision of a previous study?
YES! See; https://bugzilla.mozilla.org/show_bug.cgi?id=1492568
See previous PHD; https://docs.google.com/document/d/1mo76Ub0l5cNIII0oKqoVn25Dop4nLKKf0CVhvK4_Wps/edit

    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/webrender-performance-67/

Comment 5

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Updating 67 status to reflect the fact that this study is done.