Closed Bug 1526061 Opened 9 months ago Closed 4 months ago

[Shield] Pref-Flip Study: WebRender Performance 67, Release

Categories

(Shield :: Shield Study, enhancement)

enhancement
Not set

Tracking

(firefox67+ fixed)

RESOLVED FIXED
Tracking Status
firefox67 + fixed

People

(Reporter: experimenter, Assigned: relaas)

References

(Blocks 2 open bugs)

Details

WebRender Performance 67

We believe that enabling WebRender by default does not negatively affect crash statistics nor checkerboarding telemetry values.

More information: https://experimenter.services.mozilla.com/experiments/webrender-performance-67/

[Tracking Requested - why for this release]:

Summary: [Shield] Pref-Flip Study: WebRender Performance 67 → [Shield] Pref-Flip Study: WebRender Performance 67, Release
See Also: → 1541488

WebRender Performance 67
Targeted: Firefox Release 67.0

We have finished testing the WebRender for 67 experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • No new issues were uncovered during testing. However, we encountered 3 known issues, which are low impact and do not block the launch of the experiment.

Testing summary:

Tested Platforms:

  • Windows 10 x64 with Nvidia GT 1030
  • Windows 10 x64 with Nvidia GT 730

Tested Firefox versions:

  • Firefox Beta 67.0b15
  • Firefox Beta 67.0b18
  • Firefox Release Candidate 67 build1
  • Firefox Release Candidate 67 build2
    Experiment Type: Pref Flip Study

    What is the preference we will be changing

gfx.webrender.all.qualified

    What are the branches of the study and what values should
    each branch be set to?

- Treatment WebRender Default in Fx67 is Enabled 50%:

Value: true

In 67 Release the WebRender default setting will still be Enabled.  So our control group has it as Enabled.
        
- Treatment WebRender Test in Fx67 is a disabled state 50%:

Value: false

In 67 Release the WebRender default setting will still be Enabled.  So our branch/test group has it as Disabled.
        

    What version and channel do you intend to ship to?

5% of Release Firefox 67.0

    Are there specific criteria for participants?

Locales: All Locales

Geographic regions: ALL

Prefs: -

Studies: -

Any additional filters: normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == 'available'

There is a staged roll-out for webrender in 67 also. we would like to avoid overlap, with the rollout deploying to clients that are not enrolled in the experiment.https://bugzilla.mozilla.org/show_bug.cgi?id=1541488

    What is your intended go live date and how long will the study run?

May 21, 2019 - Jun 11, 2019 (21 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

overall crash reports ≤ 5% increase in crash rate
OOM crash reports ≤ 5% increase in crash rate
shutdown crashes ≤ 5% increase in crash rate
CANVAS_WEBGL_SUCCESS ≤ 5% regression in median of fraction "True" per user
COMPOSITE_TIME Median per-user fraction of slow frames < 0.5% (absolute)
CONTENT_FRAME_TIME_VSYNC ≤ 5% regression in median of per-user fraction of slow events
CONTENT_FULL_PAINT_TIME ≤ 5% regression in fraction of slow paints (> 16 ms), ≤ 5% regression in median of per-user means
DEVICE_RESET_REASON ≤ 5% increase in reset rate
FX_PAGE_LOAD_MS_2 ≤ 5% regression in median of per-user means
FX_TAB_SWITCH_COMPOSITE_E10S_MS ≤ 5% regression in median of per-user means

Crash (/submission/reset) rates will be measured as events per 1,000 usage hours. A "slow event" means 16 ms or 200% vsync.

    Who is the owner of the data analysis for this study?

Tim Smith

    Will this experiment require uplift?

False

    QA Status of your code:

Is this study a minor revision of a previous study?
YES! See; https://bugzilla.mozilla.org/show_bug.cgi?id=1492568
See previous PHD; https://docs.google.com/document/d/1mo76Ub0l5cNIII0oKqoVn25Dop4nLKKf0CVhvK4_Wps/edit

    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/webrender-performance-67/
    Experiment Type: Pref Flip Study

    What is the preference we will be changing

gfx.webrender.all.qualified

    What are the branches of the study and what values should
    each branch be set to?

- Treatment WebRender Default in Fx67 is Enabled 50%:

Value: true

In 67 Release the WebRender default setting will still be Enabled.  So our control group has it as Enabled.
        
- Treatment WebRender Test in Fx67 is a disabled state 50%:

Value: false

In 67 Release the WebRender default setting will still be Enabled.  So our branch/test group has it as Disabled.
        

    What version and channel do you intend to ship to?

5% of Release Firefox 67.0

    Are there specific criteria for participants?

Locales: All Locales

Geographic regions: ALL

Prefs: -

Studies: -

Any additional filters: normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == 'available'

There is a staged roll-out for webrender in 67 also. we would like to avoid overlap, with the rollout deploying to clients that are not enrolled in the experiment.https://bugzilla.mozilla.org/show_bug.cgi?id=1541488

    What is your intended go live date and how long will the study run?

May 21, 2019 - Jun 11, 2019 (21 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

overall crash reports ≤ 5% increase in crash rate
OOM crash reports ≤ 5% increase in crash rate
shutdown crashes ≤ 5% increase in crash rate
CANVAS_WEBGL_SUCCESS ≤ 5% regression in median of fraction "True" per user
COMPOSITE_TIME Median per-user fraction of slow frames < 0.5% (absolute)
CONTENT_FRAME_TIME_VSYNC ≤ 5% regression in median of per-user fraction of slow events
CONTENT_FULL_PAINT_TIME ≤ 5% regression in fraction of slow paints (> 16 ms), ≤ 5% regression in median of per-user means
DEVICE_RESET_REASON ≤ 5% increase in reset rate
FX_PAGE_LOAD_MS_2 ≤ 5% regression in median of per-user means
FX_TAB_SWITCH_COMPOSITE_E10S_MS ≤ 5% regression in median of per-user means

Crash (/submission/reset) rates will be measured as events per 1,000 usage hours. A "slow event" means 16 ms or 200% vsync.

    Who is the owner of the data analysis for this study?

Tim Smith

    Will this experiment require uplift?

False

    QA Status of your code:

Is this study a minor revision of a previous study?
YES! See; https://bugzilla.mozilla.org/show_bug.cgi?id=1492568
See previous PHD; https://docs.google.com/document/d/1mo76Ub0l5cNIII0oKqoVn25Dop4nLKKf0CVhvK4_Wps/edit

    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/webrender-performance-67/
Status: NEW → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED

Updating 67 status to reflect the fact that this study is done.

You need to log in before you can comment on or make changes to this bug.