Closed Bug 1719576 Opened 3 years ago Closed 3 years ago

[Experiment] Pref-Flip Experiment: Fission Beta 91 experiment Fx 91.0 to 91.0 Beta

Categories

(Shield :: Shield Study, task, P3)

Tracking

(firefox89 disabled, firefox90 disabled, firefox91+ fixed)

RESOLVED FIXED
Tracking Status
firefox89 --- disabled
firefox90 --- disabled
firefox91 + fixed

People

(Reporter: experimenter, Assigned: cpeterson)

References

(Depends on 1 open bug, )

Details

(Whiteboard: [fission:m7a] [no-nag])

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

fission.experiment.enrollmentStatus

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment fission-enabled 50%:

Value: 2

This branch has Fission enabled.
        
- Control fission-disabled 50%:

Value: 1

The control group does not have Fission enabled.
        

    What version and channel do you intend to ship to?

60% of Beta Firefox 91.0 to 91.0

    Are there specific criteria for participants?

Like the Fission Beta 90 experiment:

* Include both BETA and DEV EDITION channels

* Require WebRender filter from the Fission Beta 90 experiment: normandy.telemetry.main.environment.system.gfx.features.compositor == "webrender"  "webrender_software" or "webrender_software_d3d11".

UNLIKE the Fission Beta 90 experiment:

* INCLUDE Linux. NO NEED to filter Linux on the `fission.experiment.max-origins.qualified` pref like the current Fission Nightly rollout.

* Target 60% experiment (30% Fission-enabled + 30% Fission-disabled). Our Beta 90 experiment targeted 15% Fission-enabled branch, but only managed to enroll about 7% (because our experiment's one-week enrollment window coincided with Beta's 50% throttled updates during the first week of the Beta cycle). In this Beta 90 experiment, we would like to enroll about 15% Fission-enabled. Assuming we only enroll half our target, we will need a 60% experiment targeting 30% Fission-enabled treatment branch + 30% Fission-disabled control branch to actually reach 15% Fission-enabled.
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Jul 13, 2021 - Aug 10, 2021 (28 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will you use to make these decisions? What metrics are you using to measure success?

Same as our Fission Beta 90 experiment: https://experimenter.services.mozilla.com/experiments/fission-beta-90-experiment/

We want to be able to monitor the difference in stability and performance metrics between clients with and without Fission enabled. The [Fission Experiment Plan](https://docs.google.com/document/d/1ZXAHep4v1lW-ZpLMtnTXAtbXSNP8Q-gr-RjUha2Yb14/edit#heading=h.asctkf7cbxa7) has a detailed list of telemetry probes (and expectations for each probe: up, down, or flat).

We plan to monitor these metrics:

Memory metrics:

- `MEMORY_TOTAL`
- `MEMORY_UNIQUE_CONTENT_STARTUP`
- `CYCLE_COLLECTOR_MAX_PAUSE`
- `CYCLE_COLLECTOR_MAX_PAUSE_CONTENT`
- `GC_MAX_PAUSE_MS_2`
- `GC_MAX_PAUSE_MS_2_CONTENT`
- `GC_MS`
- `GC_MS_CONTENT`
- `GC_SLICE_DURING_IDLE`
- `GC_SLICE_DURING_IDLE_CONTENT`

Performance metrics:

- `CHECKERBOARDING_SEVERITY`
- `CHILD_PROCESS_LAUNCH_MS`
- `CONTENT_FRAME_TIME_VSYNC`
- `FX_NEW_WINDOW_MS`
- `FX_TAB_SWITCH_COMPOSITE_E10S_MS`
- `KEYPRESS_PRESENT_LATENCY_MS`
- `INPUT_EVENT_RESPONSE_MS`
- `TIME_TO_FIRST_CONTENTFUL_PAINT_MS`
- `TIME_TO_FIRST_INTERACTION_MS`
- `PERF_PAGE_LOAD_TIME_MS`

Engagement metrics:

- Number of users submitting pings
- `browser.engagement.total_uri_count`
- `browser.engagement.tab_open_event_count`
- `browser.engagement.max_concurrent_tab_count`
- `SUBSESSION_LENGTH`
- `CONTENT_PROCESS_COUNT`
- `CONTENT_PROCESS_MAX`
- `LOADED_TAB_COUNT`

Stability metrics:

- Main process crash rate (per 1000 usage hours), 32-bit and 64-bit
- Content process crashes (per 1000 usage hours) (excluding ShutDownKills), 32-bit and 64-bit
- `crash_tab_ui_presented`
- `crash_subframe_ui_presented`
- `oom_crashes` (per 1000 usage hours), 32-bit and 64-bit
- `shutdown_crashes` (per 1000 usage hours)

Do you plan on surveying users at the end of the delivery? No.

 Estimated Total Enrolled Clients: 360,000 (assuming we enroll 15% of 1.2M Beta + Dev Edition users running Windows or macOS)

    Who is the owner of the data analysis for this experiment?

cdowhygelund@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

NOTHING ELSE ABOUT EXPERIMENT CHANGING compared to the previous Fission Beta 90 experiment:
https://experimenter.services.mozilla.com/experiments/fission-beta-90-experiment/

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/fission-beta-91-experiment/

Fission Beta 91 experiment

Track the stability and performance effects of Fission on Beta and Dev Edition profiles

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DS-1619
More information: https://experimenter.services.mozilla.com/experiments/fission-beta-91-experiment/

Severity: -- → N/A
Whiteboard: [fission:m7a]

[Tracking Requested - why for this release]:

The Fission team would like to run yet another Fission experiment in Beta 91. We will include Linux in Beta for the first time. (We've been testing Fission on Linux in Nightly for months.)

https://experimenter.services.mozilla.com/experiments/fission-beta-91-experiment/

Depends on: 1712979
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Fission Beta 91 experiment → [Experiment] Pref-Flip Experiment: Fission Beta 91 experiment Fx 91.0 to 91.0 Beta

Based on PI-1115, we have sent a GREEN sign off.

User Story: (updated)
Start Date: 2021-07-13 End Date: 2021-08-10
Whiteboard: [fission:m7a] → [fission:m7a] [no-nag]
Blocks: 1724054
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.