Closed Bug 1526061 Opened 6 years ago Closed 5 years ago

[Shield] Pref-Flip Study: WebRender Performance 67, Release

Categories

(Shield :: Shield Study, enhancement)

Product:
Shield
Component:
Shield Study
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox67+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox67 + fixed

People

(Reporter: experimenter, Assigned: relaas)

References

Details

WebRender Performance 67

We believe that enabling WebRender by default does not negatively affect crash statistics nor checkerboarding telemetry values.

More information: https://experimenter.services.mozilla.com/experiments/webrender-performance-67/

[Tracking Requested - why for this release]:

Blocks: stage-wr-trains
tracking-firefox67: --- → ?
Summary: [Shield] Pref-Flip Study: WebRender Performance 67 → [Shield] Pref-Flip Study: WebRender Performance 67, Release

WebRender Performance 67
Targeted: Firefox Release 67.0

We have finished testing the WebRender for 67 experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • No new issues were uncovered during testing. However, we encountered 3 known issues, which are low impact and do not block the launch of the experiment.

Testing summary:

Tested Platforms:

  • Windows 10 x64 with Nvidia GT 1030
  • Windows 10 x64 with Nvidia GT 730

Tested Firefox versions:

  • Firefox Beta 67.0b15
  • Firefox Beta 67.0b18
  • Firefox Release Candidate 67 build1
  • Firefox Release Candidate 67 build2
Experiment Type: Pref Flip Study What is the preference we will be changing gfx.webrender.all.qualified What are the branches of the study and what values should each branch be set to? - Treatment WebRender Default in Fx67 is Enabled 50%: Value: true In 67 Release the WebRender default setting will still be Enabled. So our control group has it as Enabled. - Treatment WebRender Test in Fx67 is a disabled state 50%: Value: false In 67 Release the WebRender default setting will still be Enabled. So our branch/test group has it as Disabled. What version and channel do you intend to ship to? 5% of Release Firefox 67.0 Are there specific criteria for participants? Locales: All Locales Geographic regions: ALL Prefs: - Studies: - Any additional filters: normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == 'available' There is a staged roll-out for webrender in 67 also. we would like to avoid overlap, with the rollout deploying to clients that are not enrolled in the experiment.https://bugzilla.mozilla.org/show_bug.cgi?id=1541488 What is your intended go live date and how long will the study run? May 21, 2019 - Jun 11, 2019 (21 days) What is the main effect you are looking for and what data will you use to make these decisions? overall crash reports ≤ 5% increase in crash rate OOM crash reports ≤ 5% increase in crash rate shutdown crashes ≤ 5% increase in crash rate CANVAS_WEBGL_SUCCESS ≤ 5% regression in median of fraction "True" per user COMPOSITE_TIME Median per-user fraction of slow frames < 0.5% (absolute) CONTENT_FRAME_TIME_VSYNC ≤ 5% regression in median of per-user fraction of slow events CONTENT_FULL_PAINT_TIME ≤ 5% regression in fraction of slow paints (> 16 ms), ≤ 5% regression in median of per-user means DEVICE_RESET_REASON ≤ 5% increase in reset rate FX_PAGE_LOAD_MS_2 ≤ 5% regression in median of per-user means FX_TAB_SWITCH_COMPOSITE_E10S_MS ≤ 5% regression in median of per-user means Crash (/submission/reset) rates will be measured as events per 1,000 usage hours. A "slow event" means 16 ms or 200% vsync. Who is the owner of the data analysis for this study? Tim Smith Will this experiment require uplift? False QA Status of your code: Is this study a minor revision of a previous study? YES! See; https://bugzilla.mozilla.org/show_bug.cgi?id=1492568 See previous PHD; https://docs.google.com/document/d/1mo76Ub0l5cNIII0oKqoVn25Dop4nLKKf0CVhvK4_Wps/edit Link to more information about this study: https://experimenter.services.mozilla.com/experiments/webrender-performance-67/
Experiment Type: Pref Flip Study What is the preference we will be changing gfx.webrender.all.qualified What are the branches of the study and what values should each branch be set to? - Treatment WebRender Default in Fx67 is Enabled 50%: Value: true In 67 Release the WebRender default setting will still be Enabled. So our control group has it as Enabled. - Treatment WebRender Test in Fx67 is a disabled state 50%: Value: false In 67 Release the WebRender default setting will still be Enabled. So our branch/test group has it as Disabled. What version and channel do you intend to ship to? 5% of Release Firefox 67.0 Are there specific criteria for participants? Locales: All Locales Geographic regions: ALL Prefs: - Studies: - Any additional filters: normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == 'available' There is a staged roll-out for webrender in 67 also. we would like to avoid overlap, with the rollout deploying to clients that are not enrolled in the experiment.https://bugzilla.mozilla.org/show_bug.cgi?id=1541488 What is your intended go live date and how long will the study run? May 21, 2019 - Jun 11, 2019 (21 days) What is the main effect you are looking for and what data will you use to make these decisions? overall crash reports ≤ 5% increase in crash rate OOM crash reports ≤ 5% increase in crash rate shutdown crashes ≤ 5% increase in crash rate CANVAS_WEBGL_SUCCESS ≤ 5% regression in median of fraction "True" per user COMPOSITE_TIME Median per-user fraction of slow frames < 0.5% (absolute) CONTENT_FRAME_TIME_VSYNC ≤ 5% regression in median of per-user fraction of slow events CONTENT_FULL_PAINT_TIME ≤ 5% regression in fraction of slow paints (> 16 ms), ≤ 5% regression in median of per-user means DEVICE_RESET_REASON ≤ 5% increase in reset rate FX_PAGE_LOAD_MS_2 ≤ 5% regression in median of per-user means FX_TAB_SWITCH_COMPOSITE_E10S_MS ≤ 5% regression in median of per-user means Crash (/submission/reset) rates will be measured as events per 1,000 usage hours. A "slow event" means 16 ms or 200% vsync. Who is the owner of the data analysis for this study? Tim Smith Will this experiment require uplift? False QA Status of your code: Is this study a minor revision of a previous study? YES! See; https://bugzilla.mozilla.org/show_bug.cgi?id=1492568 See previous PHD; https://docs.google.com/document/d/1mo76Ub0l5cNIII0oKqoVn25Dop4nLKKf0CVhvK4_Wps/edit Link to more information about this study: https://experimenter.services.mozilla.com/experiments/webrender-performance-67/
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED

Updating 67 status to reflect the fact that this study is done.

status-firefox67: affected → fixed
You need to log in before you can comment on or make changes to this bug.