Closed Bug 1731908 Opened 3 years ago Closed 3 years ago

[Experiment] Pref-Flip Experiment: Fission Beta 94-95 experiment Fx 94.0 to 95.0 Beta

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Platform:
Unspecified
All
Type:
task

Tracking

(firefox92 wontfix, firefox93 wontfix, firefox94 affected)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox92 --- wontfix
firefox93 --- wontfix
firefox94 --- affected

People

(Reporter: experimenter, Assigned: cpeterson)

References

(Blocks 1 open bug,
URL
)

Details

(Whiteboard: [fission:mvp])

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

fission.experiment.enrollmentStatus

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment fission-enabled 50%:

Value: 2

This branch has Fission enabled.
        
- Control fission-disabled 50%:

Value: 1

The control group does not have Fission enabled.
        

    What version and channel do you intend to ship to?

100% of Beta Firefox 94.0 to 95.0

    Are there specific criteria for participants?

Like the Fission Beta 91 experiment:

* Include both BETA and DEV EDITION channels
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Oct 05, 2021 - Nov 30, 2021 (56 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will you use to make these decisions? What metrics are you using to measure success?

Same as our Fission Beta 91 experiment: https://experimenter.services.mozilla.com/experiments/fission-beta-91-experiment/

We want to be able to monitor the difference in stability and performance metrics between clients with and without Fission enabled. The [Fission Experiment Plan](https://docs.google.com/document/d/1ZXAHep4v1lW-ZpLMtnTXAtbXSNP8Q-gr-RjUha2Yb14/edit#heading=h.asctkf7cbxa7) has a detailed list of telemetry probes (and expectations for each probe: up, down, or flat).

We plan to monitor these metrics:

Memory metrics:

- `MEMORY_TOTAL`
- `MEMORY_UNIQUE_CONTENT_STARTUP`
- `CYCLE_COLLECTOR_MAX_PAUSE`
- `CYCLE_COLLECTOR_MAX_PAUSE_CONTENT`
- `GC_MAX_PAUSE_MS_2`
- `GC_MAX_PAUSE_MS_2_CONTENT`
- `GC_MS`
- `GC_MS_CONTENT`
- `GC_SLICE_DURING_IDLE`
- `GC_SLICE_DURING_IDLE_CONTENT`

Performance metrics:

- `CHECKERBOARDING_SEVERITY`
- `CHILD_PROCESS_LAUNCH_MS`
- `CONTENT_FRAME_TIME_VSYNC`
- `FX_NEW_WINDOW_MS`
- `FX_TAB_SWITCH_COMPOSITE_E10S_MS`
- `KEYPRESS_PRESENT_LATENCY_MS`
- `INPUT_EVENT_RESPONSE_MS`
- `TIME_TO_FIRST_CONTENTFUL_PAINT_MS`
- `TIME_TO_FIRST_INTERACTION_MS`
- `PERF_PAGE_LOAD_TIME_MS`

Engagement metrics:

- Number of users submitting pings
- `browser.engagement.total_uri_count`
- `browser.engagement.tab_open_event_count`
- `browser.engagement.max_concurrent_tab_count`
- `SUBSESSION_LENGTH`
- `CONTENT_PROCESS_COUNT`
- `CONTENT_PROCESS_MAX`
- `LOADED_TAB_COUNT`

Stability metrics:

- Main process crash rate (per 1000 usage hours), 32-bit and 64-bit
- Content process crashes (per 1000 usage hours) (excluding ShutDownKills), 32-bit and 64-bit
- `crash_tab_ui_presented`
- `crash_subframe_ui_presented`
- `oom_crashes` (per 1000 usage hours), 32-bit and 64-bit
- `shutdown_crashes` (per 1000 usage hours)

Do you plan on surveying users at the end of the delivery? No.

Estimated Total Enrolled Clients: 1,200,000. Like this experiment, our Fission Beta 92 experiment targeted 100% of Beta and Dev Edition users on Windows, macOS, and Linux. The final enrollment for Beta 92 was 1,252,000 (625K Fission treatment branch + 627K non-Fission control branch).

    Who is the owner of the data analysis for this experiment?

None

    Will this experiment require uplift?

False

    QA Status of your code:

NOTHING ELSE ABOUT EXPERIMENT CHANGING compared to the previous Fission Beta 92 experiment:
https://experimenter.services.mozilla.com/experiments/fission-beta-92-experiment/

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/fission-beta-94-95-experiment/

Fission Beta 94-95 experiment

Track the stability and performance effects of Fission on Beta and Dev Edition profiles

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://mozilla-hub.atlassian.net/browse/DO-601
More information: https://experimenter.services.mozilla.com/experiments/fission-beta-94-95-experiment/

status-firefox92: --- → wontfix
status-firefox93: --- → wontfix
status-firefox94: --- → affected
OS: Unspecified → All
Whiteboard: [fission:mvp]
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Fission Beta 94-95 experiment → [Experiment] Pref-Flip Experiment: Fission Beta 94-95 experiment Fx 94.0 to 95.0 Beta
Start Date: 2021-10-05 End Date: 2021-11-30
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.