[Shield] Pref-Flip Study: WebRender Performance 66, 66, Release

RESOLVED FIXED

Status

enhancement
RESOLVED FIXED
4 months ago
2 months ago

People

(Reporter: experimenter, Assigned: relaas)

Tracking

(Blocks 1 bug)

unspecified
Dependency tree / graph
Bug Flags:
shield-qa +

Firefox Tracking Flags

(firefox66- affected)

Details

Reporter

Description

4 months ago

WebRender Performance 66

We believe that enabling WebRender by default does not negatively affect crash statistics nor checkerboarding telemetry values.

More information: https://experimenter.services.mozilla.com/experiments/webrender-performance-66/

Comment 1

4 months ago

[Tracking Requested - why for this release]:

Summary: [Shield] Pref-Flip Study: WebRender Performance 66 → [Shield] Pref-Flip Study: WebRender Performance 66, 66, Release

Updated

4 months ago

Subject: [Shield Study] Sign Off for WebRender Performance 66 - (YELLOW)

WebRender Performance 66
Targeted: Firefox Release 66.0

We have finished testing the WebRender for 66 experiment.

QA’s recommendation: YELLOW - SHIP IT, CONDITIONALLY

Reasoning:

  • During testing we found two new issues and came across several known issues.
  • A couple of issues are fixed and verified on Nightly 67, but are marked as disabled for Firefox 66.

Testing summary:

Tested Platforms:

  • Windows 10 x64 with Nvidia GT 1030
  • Windows 10 x64 with Nvidia GT 730

Tested Firefox versions:

  • Firefox Beta 66.0b13
  • Firefox Beta 66.0b14
  • Firefox Release Candidate 66.0
Flags: shield-qa+
Reporter

Comment 3

3 months ago
    Experiment Type: Pref Flip Study

    What is the preference we will be changing

gfx.webrender.all.qualified

    What are the branches of the study and what values should
    each branch be set to?

- Treatment WebRender Test in 66 is Enabled 50%:

Value: true

In 66, the test group will be to enable WebRender.
        
- Treatment WebRender Default in 66 is Disabled 50%:

Value: false

In 66 Release the WebRender default setting will still be Disabled.  So our control group has it as Disabled.
        

    What version and channel do you intend to ship to?

1.5% of Release Firefox 66.0

    Are there specific criteria for participants?

Locales: All

Geographic regions: All

Prefs: -

Studies: -

Any additional filters: normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == 'available'

About 5% of users will meet the wrQualified filter condition, so about 0.075% of clients will actually enroll with a 1.5% filter on client_id.

Power analysis described at https://bugzilla.mozilla.org/show_bug.cgi?id=1526041

    What is your intended go live date and how long will the study run?

Mar 25, 2019 - Apr 15, 2019 (21 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

overall crash reports 	≤ 5% increase in crash rate
OOM crash reports 	≤ 5% increase in crash rate
shutdown crashes 	≤ 5% increase in crash rate
CANVAS_WEBGL_SUCCESS 	≤ 5% regression in median of fraction "True" per user
COMPOSITE_TIME 	Median per-user fraction of slow frames < 0.5% (absolute)
CONTENT_FRAME_TIME_VSYNC 	≤ 5% regression in median of per-user fraction of slow events
CONTENT_FULL_PAINT_TIME 	≤ 5% regression in fraction of slow paints (> 16 ms), ≤ 5% regression in median of per-user means
DEVICE_RESET_REASON 	≤ 5% increase in reset rate
FX_PAGE_LOAD_MS_2 	≤ 5% regression in median of per-user means
FX_TAB_SWITCH_COMPOSITE_E10S_MS 	≤ 5% regression in median of per-user means

Crash (/submission/reset) rates will be measured as events per 1,000 usage hours. A "slow event" means 16 ms or 200% vsync.

    Who is the owner of the data analysis for this study?

tdsmith

    Will this experiment require uplift?

False

    QA Status of your code:

This study is a revision of a currently running study
https://bugzilla.mozilla.org/show_bug.cgi?id=1492568

This aren't any code changes - just launching the same study in release.

    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/webrender-performance-66/
Reporter

Comment 4

3 months ago
    Experiment Type: Pref Flip Study

    What is the preference we will be changing

gfx.webrender.all.qualified

    What are the branches of the study and what values should
    each branch be set to?

- Treatment WebRender Test in 66 is Enabled 50%:

Value: true

In 66, the test group will be to enable WebRender.
        
- Treatment WebRender Default in 66 is Disabled 50%:

Value: false

In 66 Release the WebRender default setting will still be Disabled.  So our control group has it as Disabled.
        

    What version and channel do you intend to ship to?

1.5% of Release Firefox 66.0

    Are there specific criteria for participants?

Locales: All

Geographic regions: All

Prefs: -

Studies: -

Any additional filters: normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == 'available'

About 5% of users will meet the wrQualified filter condition, so about 0.075% of clients will actually enroll with a 1.5% filter on client_id.

Power analysis described at https://bugzilla.mozilla.org/show_bug.cgi?id=1526041

    What is your intended go live date and how long will the study run?

Mar 25, 2019 - Apr 15, 2019 (21 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

overall crash reports 	≤ 5% increase in crash rate
OOM crash reports 	≤ 5% increase in crash rate
shutdown crashes 	≤ 5% increase in crash rate
CANVAS_WEBGL_SUCCESS 	≤ 5% regression in median of fraction "True" per user
COMPOSITE_TIME 	Median per-user fraction of slow frames < 0.5% (absolute)
CONTENT_FRAME_TIME_VSYNC 	≤ 5% regression in median of per-user fraction of slow events
CONTENT_FULL_PAINT_TIME 	≤ 5% regression in fraction of slow paints (> 16 ms), ≤ 5% regression in median of per-user means
DEVICE_RESET_REASON 	≤ 5% increase in reset rate
FX_PAGE_LOAD_MS_2 	≤ 5% regression in median of per-user means
FX_TAB_SWITCH_COMPOSITE_E10S_MS 	≤ 5% regression in median of per-user means

Crash (/submission/reset) rates will be measured as events per 1,000 usage hours. A "slow event" means 16 ms or 200% vsync.

    Who is the owner of the data analysis for this study?

tdsmith

    Will this experiment require uplift?

False

    QA Status of your code:

This study is a revision of a currently running study
https://bugzilla.mozilla.org/show_bug.cgi?id=1492568

This aren't any code changes - just launching the same study in release.

    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/webrender-performance-66/

Untracking since we have this information easily findable now in Experimenter.

The experiment has ended; followup in bug 1521626.

Status: NEW → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.