Closed Bug 1526060 Opened 6 years ago Closed 6 years ago

[Shield] Pref-Flip Study: WebRender Performance 66, 66, Release

Categories

(Shield :: Shield Study, enhancement)

Product:
Shield
Component:
Shield Study
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox66- affected)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox66 - affected

People

(Reporter: experimenter, Assigned: relaas)

References

Details

WebRender Performance 66

We believe that enabling WebRender by default does not negatively affect crash statistics nor checkerboarding telemetry values.

More information: https://experimenter.services.mozilla.com/experiments/webrender-performance-66/

[Tracking Requested - why for this release]:

tracking-firefox66: --- → ?
Summary: [Shield] Pref-Flip Study: WebRender Performance 66 → [Shield] Pref-Flip Study: WebRender Performance 66, 66, Release
Blocks: stage-wr-trains

Subject: [Shield Study] Sign Off for WebRender Performance 66 - (YELLOW)

WebRender Performance 66
Targeted: Firefox Release 66.0

We have finished testing the WebRender for 66 experiment.

QA’s recommendation: YELLOW - SHIP IT, CONDITIONALLY

Reasoning:

  • During testing we found two new issues and came across several known issues.
  • A couple of issues are fixed and verified on Nightly 67, but are marked as disabled for Firefox 66.

Testing summary:

Tested Platforms:

  • Windows 10 x64 with Nvidia GT 1030
  • Windows 10 x64 with Nvidia GT 730

Tested Firefox versions:

  • Firefox Beta 66.0b13
  • Firefox Beta 66.0b14
  • Firefox Release Candidate 66.0
Flags: shield-qa+
Experiment Type: Pref Flip Study What is the preference we will be changing gfx.webrender.all.qualified What are the branches of the study and what values should each branch be set to? - Treatment WebRender Test in 66 is Enabled 50%: Value: true In 66, the test group will be to enable WebRender. - Treatment WebRender Default in 66 is Disabled 50%: Value: false In 66 Release the WebRender default setting will still be Disabled. So our control group has it as Disabled. What version and channel do you intend to ship to? 1.5% of Release Firefox 66.0 Are there specific criteria for participants? Locales: All Geographic regions: All Prefs: - Studies: - Any additional filters: normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == 'available' About 5% of users will meet the wrQualified filter condition, so about 0.075% of clients will actually enroll with a 1.5% filter on client_id. Power analysis described at https://bugzilla.mozilla.org/show_bug.cgi?id=1526041 What is your intended go live date and how long will the study run? Mar 25, 2019 - Apr 15, 2019 (21 days) What is the main effect you are looking for and what data will you use to make these decisions? overall crash reports ≤ 5% increase in crash rate OOM crash reports ≤ 5% increase in crash rate shutdown crashes ≤ 5% increase in crash rate CANVAS_WEBGL_SUCCESS ≤ 5% regression in median of fraction "True" per user COMPOSITE_TIME Median per-user fraction of slow frames < 0.5% (absolute) CONTENT_FRAME_TIME_VSYNC ≤ 5% regression in median of per-user fraction of slow events CONTENT_FULL_PAINT_TIME ≤ 5% regression in fraction of slow paints (> 16 ms), ≤ 5% regression in median of per-user means DEVICE_RESET_REASON ≤ 5% increase in reset rate FX_PAGE_LOAD_MS_2 ≤ 5% regression in median of per-user means FX_TAB_SWITCH_COMPOSITE_E10S_MS ≤ 5% regression in median of per-user means Crash (/submission/reset) rates will be measured as events per 1,000 usage hours. A "slow event" means 16 ms or 200% vsync. Who is the owner of the data analysis for this study? tdsmith Will this experiment require uplift? False QA Status of your code: This study is a revision of a currently running study https://bugzilla.mozilla.org/show_bug.cgi?id=1492568 This aren't any code changes - just launching the same study in release. Link to more information about this study: https://experimenter.services.mozilla.com/experiments/webrender-performance-66/
Experiment Type: Pref Flip Study What is the preference we will be changing gfx.webrender.all.qualified What are the branches of the study and what values should each branch be set to? - Treatment WebRender Test in 66 is Enabled 50%: Value: true In 66, the test group will be to enable WebRender. - Treatment WebRender Default in 66 is Disabled 50%: Value: false In 66 Release the WebRender default setting will still be Disabled. So our control group has it as Disabled. What version and channel do you intend to ship to? 1.5% of Release Firefox 66.0 Are there specific criteria for participants? Locales: All Geographic regions: All Prefs: - Studies: - Any additional filters: normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == 'available' About 5% of users will meet the wrQualified filter condition, so about 0.075% of clients will actually enroll with a 1.5% filter on client_id. Power analysis described at https://bugzilla.mozilla.org/show_bug.cgi?id=1526041 What is your intended go live date and how long will the study run? Mar 25, 2019 - Apr 15, 2019 (21 days) What is the main effect you are looking for and what data will you use to make these decisions? overall crash reports ≤ 5% increase in crash rate OOM crash reports ≤ 5% increase in crash rate shutdown crashes ≤ 5% increase in crash rate CANVAS_WEBGL_SUCCESS ≤ 5% regression in median of fraction "True" per user COMPOSITE_TIME Median per-user fraction of slow frames < 0.5% (absolute) CONTENT_FRAME_TIME_VSYNC ≤ 5% regression in median of per-user fraction of slow events CONTENT_FULL_PAINT_TIME ≤ 5% regression in fraction of slow paints (> 16 ms), ≤ 5% regression in median of per-user means DEVICE_RESET_REASON ≤ 5% increase in reset rate FX_PAGE_LOAD_MS_2 ≤ 5% regression in median of per-user means FX_TAB_SWITCH_COMPOSITE_E10S_MS ≤ 5% regression in median of per-user means Crash (/submission/reset) rates will be measured as events per 1,000 usage hours. A "slow event" means 16 ms or 200% vsync. Who is the owner of the data analysis for this study? tdsmith Will this experiment require uplift? False QA Status of your code: This study is a revision of a currently running study https://bugzilla.mozilla.org/show_bug.cgi?id=1492568 This aren't any code changes - just launching the same study in release. Link to more information about this study: https://experimenter.services.mozilla.com/experiments/webrender-performance-66/

Untracking since we have this information easily findable now in Experimenter.

tracking-firefox66: + → -

The experiment has ended; followup in bug 1521626.

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.