Closed Bug 1660366 (fission-nightly-experiment) Opened 4 years ago Closed 3 years ago

[Experiment] Pref-Flip Experiment: Ongoing Fission Nightly experiment Fx 83.0 to 100.0 Nightly

Tracking

(firefox81 wontfix, firefox82 wontfix, firefox83 affected)

Status:

RESOLVED FIXED

Tracking Flags:

Tracking

Status

firefox81

---

wontfix

firefox82

---

wontfix

firefox83

---

affected

People

(Reporter: experimenter, Assigned: cpeterson)

References

(Blocks 1 open bug,
URL
)

Details

(Whiteboard: [fission:m6b])

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

fission.experiment.enrollmentStatus

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment fission-enabled 50%:

Value: 2

This branch has Fission enabled.
        
- Control fission-disabled 50%:

Value: 1

The control group does not have Fission enabled.
        

    What version and channel do you intend to ship to?

5% of Nightly Firefox 83.0 to 100.0

    Are there specific criteria for participants?

Filters:

Exclude users with Fission already configured (user branch value): `fission.autostart` pref has USER value. We don't want users who are already dogfooding Fission to be part of this experiment.

- Require WebRender for Fission on Windows and macOS for our Nightly experiment. Users must be WebRender "qualified": `normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == ‘available’`. (We hope to eventually ship support for Fission with or without WebRender, but we will require it now to simplify our Nightly experiment.)

- Exclude users in any WebRender experiments. The current WebRender v2 experiment (https://experimenter.services.mozilla.com/experiments/webrender-continued-v2-nightly-only/) slug is “bug-1622934-pref-webrender-continued-v2-nightly-only-nightly-76-80”.

The Fission Experiment Plan (https://docs.google.com/document/d/1ZXAHep4v1lW-ZpLMtnTXAtbXSNP8Q-gr-RjUha2Yb14/edit#heading=h.dj0tcj3def9) has more detailed explanations of these eligibility criteria.

Corey's Colab notebook estimating we will need 5% sample size to get 1000 users/branch: https://sql.telemetry.mozilla.org/queries/74160/source
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Oct 06, 2020 - Oct 06, 2021 (365 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will you use to make these decisions? What metrics are you using to measure success?

We want to be able to monitor the difference in stability and performance metrics between clients with and without Fission enabled. The [Fission Experiment Plan](https://docs.google.com/document/d/1ZXAHep4v1lW-ZpLMtnTXAtbXSNP8Q-gr-RjUha2Yb14/edit#heading=h.asctkf7cbxa7) has a detailed list of telemetry probes (and expectations for each probe: up, down, or flat).

We plan to monitor these existing metrics:

- `CHECKERBOARDING_SEVERITY`
- `CHILD_PROCESS_LAUNCH_MS`
- `CONTENT_FRAME_TIME_VSYNC`
- `SCALARS_GFX.OMTP.PAINT_WAIT_RATIO`
- `FX_NEW_WINDOW_MS`
- `FX_TAB_SWITCH_COMPOSITE_E10S_MS`
- `KEYPRESS_PRESENT_LATENCY_MS`
- `INPUT_EVENT_RESPONSE_MS`
- `MEMORY_TOTAL`
- `CYCLE_COLLECTOR_MAX_PAUSE`
- `GC_MAX_PAUSE_MS_2`
- `GC_MS`
- `GC_SLICE_DURING_IDLE`
- `MEMORY_UNIQUE_CONTENT_STARTUP`
- `FX_NUMBER_OF_UNIQUE_SITE_ORIGINS_PER_LOADED_TABS`
- `FX_NUMBER_OF_UNIQUE_SITE_ORIGINS_ALL_TABS`
- `TIME_TO_FIRST_CONTENTFUL_PAINT_MS`
- `TIME_TO_FIRST_INTERACTION_MS`
- `FX_PAGE_LOAD_MS_2`
- `LOADED_TAB_COUNT`
- `browser.engagement.total_uri_count`
- `browser.engagement.tab_open_event_count`
- `browser.engagement.max_concurrent_tab_count`
- Number of users submitting pings
- Main process crash rate (per 1000 usage hours), 32-bit and 64-bit
- Content process crashes (per 1000 usage hours) (excluding ShutdownKills), 32-bit and 64-bit
- `oom_crashes` (per 1000 usage hours), 32-bit and 64-bit
- `shutdown_crashes` (per 1000 usage hours)
- GPU process crashes (per 1000 usage hours)

Do you plan on surveying users at the end of the delivery? No.

 Estimated Total Enrolled Clients: 2500 (assuming we enroll 5% of ~50,000 Nightly users)

    Who is the owner of the data analysis for this experiment?

cdowhygelund@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

Not started

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/fission-nightly/

Experimenter Bot

Reporter

Description

•

4 years ago

Ongoing Fission Nightly experiment

Track the stability and performance effects of Fission

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DS-753
More information: https://experimenter.services.mozilla.com/experiments/fission-nightly/

Chris Peterson [:cpeterson]

Assignee

Updated

•

4 years ago

Alias: fission-nightly-experiment

Severity: -- → N/A

Whiteboard: [fission:mb6]

Chris Peterson [:cpeterson]

Assignee

Comment 1

•

4 years ago

We are aiming to launch our Fission Nightly experiment on Oct 6 (Fx83 Nightly).

status-firefox81: --- → wontfix

status-firefox82: --- → wontfix

status-firefox83: --- → affected

Experimenter Bot

Reporter

Updated

•

4 years ago

User Story: (updated)

Summary: [Experiment]: Pref-Flip Experiment: Ongoing Fission Nightly experiment → [Experiment] Pref-Flip Experiment: Ongoing Fission Nightly experiment Fx 83.0 to 100.0 Nightly

Chris Peterson [:cpeterson]

Assignee

Updated

•

4 years ago

User Story: (updated)

Whiteboard: [fission:mb6] → [fission:m6b]

Experimenter Bot

Reporter

Comment 2

•

4 years ago

Start Date: 2020-09-29 End Date: 2021-09-29

We have finished testing the Ongoing Fission Nightly experiment experiment.

Quality status: YELLOW - SHIP IT CONDITIONALLY

Why is this experiment yellow?

During testing we found 7 issues, 3 of which are still open. They are not blocking, but one of the open issues has been found late in the testing process and needs the assessment of the Engineering team:
Bug 1670263 - The Fission doorhanger is dismissed by some dialogs
An enhancement has recently been opened, Bug 1670587, that still needs to land and to be verified by QA.

What needs to be done?

[Engineering] The team needs to assess whether Bug 1670263 needs to be fixed before launch of the experiment based on how often it is likely to be encountered.
[QA] We will need to verify Bug 1670263 in the case it is fixed.
[QA] We will verify Bug 1670587 when it has landed.

Testing Summary:

Test runs: TestRail.
Exploratory testing related to:
enrollment and unenrollment;
the doorhanger;
the interaction between the experiment and the addition of Fission to the list of experimental features found in “about:preferences#experimental” introduced in Bug 1652134.

Tested Platforms:

Windows 10 x64 Intel/Nvidia GPU
Windows 7 AMD/Intel GPU
macOS 10.15 Nvidia/Intel GPU

Tested Firefox versions:

Firefox Nightly 83.0a1 (Build IDs 20201004212809 through 20201009041754)

Regards,
Maria
Ecosystem QA Team

Chris Peterson [:cpeterson]

Assignee

Comment 4

•

4 years ago

(In reply to Maria Heres, :mheres, Ecosystem QA from comment #3)

Quality status: YELLOW - SHIP IT CONDITIONALLY

Why is this experiment yellow?

During testing we found 7 issues, 3 of which are still open. They are not blocking, but one of the open issues has been found late in the testing process and needs the assessment of the Engineering team:

Bug 1670263 - The Fission doorhanger is dismissed by some dialogs

Fixing bug 1670263 would be nice (if it doesn't require a lot of custom code to work around the expected doorhanger behavior), but I don't think that bug needs to block launching our Fission experiment.

Chris Peterson [:cpeterson]

Assignee