Open Bug 1660366 (fission-nightly-experiment) Opened 7 months ago Updated 5 months ago

[Experiment] Pref-Flip Experiment: Ongoing Fission Nightly experiment Fx 83.0 to 100.0 Nightly

Categories

(Shield :: Shield Study, task, P3)

Tracking

(firefox81 wontfix, firefox82 wontfix, firefox83 affected)

Tracking Status
firefox81 --- wontfix
firefox82 --- wontfix
firefox83 --- affected

People

(Reporter: experimenter, Assigned: cpeterson)

References

(Depends on 1 open bug, Blocks 1 open bug, )

Details

(Whiteboard: [fission:m6b])

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

fission.experiment.enrollmentStatus

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment fission-enabled 50%:

Value: 2

This branch has Fission enabled.
        
- Control fission-disabled 50%:

Value: 1

The control group does not have Fission enabled.
        

    What version and channel do you intend to ship to?

5% of Nightly Firefox 83.0 to 100.0

    Are there specific criteria for participants?

Filters:

Exclude users with Fission already configured (user branch value): `fission.autostart` pref has USER value. We don't want users who are already dogfooding Fission to be part of this experiment.

- Require WebRender for Fission on Windows and macOS for our Nightly experiment. Users must be WebRender "qualified": `normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == ‘available’`. (We hope to eventually ship support for Fission with or without WebRender, but we will require it now to simplify our Nightly experiment.)

- Exclude users in any WebRender experiments. The current WebRender v2 experiment (https://experimenter.services.mozilla.com/experiments/webrender-continued-v2-nightly-only/) slug is “bug-1622934-pref-webrender-continued-v2-nightly-only-nightly-76-80”.

The Fission Experiment Plan (https://docs.google.com/document/d/1ZXAHep4v1lW-ZpLMtnTXAtbXSNP8Q-gr-RjUha2Yb14/edit#heading=h.dj0tcj3def9) has more detailed explanations of these eligibility criteria.

Corey's Colab notebook estimating we will need 5% sample size to get 1000 users/branch: https://sql.telemetry.mozilla.org/queries/74160/source
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Oct 06, 2020 - Oct 06, 2021 (365 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will you use to make these decisions? What metrics are you using to measure success?

We want to be able to monitor the difference in stability and performance metrics between clients with and without Fission enabled. The [Fission Experiment Plan](https://docs.google.com/document/d/1ZXAHep4v1lW-ZpLMtnTXAtbXSNP8Q-gr-RjUha2Yb14/edit#heading=h.asctkf7cbxa7) has a detailed list of telemetry probes (and expectations for each probe: up, down, or flat).

We plan to monitor these existing metrics:

- `CHECKERBOARDING_SEVERITY`
- `CHILD_PROCESS_LAUNCH_MS`
- `CONTENT_FRAME_TIME_VSYNC`
- `SCALARS_GFX.OMTP.PAINT_WAIT_RATIO`
- `FX_NEW_WINDOW_MS`
- `FX_TAB_SWITCH_COMPOSITE_E10S_MS`
- `KEYPRESS_PRESENT_LATENCY_MS`
- `INPUT_EVENT_RESPONSE_MS`
- `MEMORY_TOTAL`
- `CYCLE_COLLECTOR_MAX_PAUSE`
- `GC_MAX_PAUSE_MS_2`
- `GC_MS`
- `GC_SLICE_DURING_IDLE`
- `MEMORY_UNIQUE_CONTENT_STARTUP`
- `FX_NUMBER_OF_UNIQUE_SITE_ORIGINS_PER_LOADED_TABS`
- `FX_NUMBER_OF_UNIQUE_SITE_ORIGINS_ALL_TABS`
- `TIME_TO_FIRST_CONTENTFUL_PAINT_MS`
- `TIME_TO_FIRST_INTERACTION_MS`
- `FX_PAGE_LOAD_MS_2`
- `LOADED_TAB_COUNT`
- `browser.engagement.total_uri_count`
- `browser.engagement.tab_open_event_count`
- `browser.engagement.max_concurrent_tab_count`
- Number of users submitting pings
- Main process crash rate (per 1000 usage hours), 32-bit and 64-bit
- Content process crashes (per 1000 usage hours) (excluding ShutdownKills), 32-bit and 64-bit
- `oom_crashes` (per 1000 usage hours), 32-bit and 64-bit
- `shutdown_crashes` (per 1000 usage hours)
- GPU process crashes (per 1000 usage hours)

Do you plan on surveying users at the end of the delivery? No.

 Estimated Total Enrolled Clients: 2500 (assuming we enroll 5% of ~50,000 Nightly users)

    Who is the owner of the data analysis for this experiment?

cdowhygelund@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

Not started

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/fission-nightly/

Ongoing Fission Nightly experiment

Track the stability and performance effects of Fission

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DS-753
More information: https://experimenter.services.mozilla.com/experiments/fission-nightly/

Alias: fission-nightly-experiment
Severity: -- → N/A
Whiteboard: [fission:mb6]

We are aiming to launch our Fission Nightly experiment on Oct 6 (Fx83 Nightly).

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Ongoing Fission Nightly experiment → [Experiment] Pref-Flip Experiment: Ongoing Fission Nightly experiment Fx 83.0 to 100.0 Nightly
User Story: (updated)
Whiteboard: [fission:mb6] → [fission:m6b]
Start Date: 2020-09-29 End Date: 2021-09-29

Ongoing Fission Nightly experiment
Nightly Firefox 83.0 to 100.0

We have finished testing the Ongoing Fission Nightly experiment experiment.

Quality status: YELLOW - SHIP IT CONDITIONALLY

Why is this experiment yellow?

  • During testing we found 7 issues, 3 of which are still open. They are not blocking, but one of the open issues has been found late in the testing process and needs the assessment of the Engineering team:

  • Bug 1670263 - The Fission doorhanger is dismissed by some dialogs

  • An enhancement has recently been opened, Bug 1670587, that still needs to land and to be verified by QA.

What needs to be done?

  • [Engineering] The team needs to assess whether Bug 1670263 needs to be fixed before launch of the experiment based on how often it is likely to be encountered.

  • [QA] We will need to verify Bug 1670263 in the case it is fixed.

  • [QA] We will verify Bug 1670587 when it has landed.

Testing Summary:

  • Test runs: TestRail.

  • Exploratory testing related to:

  • enrollment and unenrollment;

  • the doorhanger;

  • the interaction between the experiment and the addition of Fission to the list of experimental features found in “about:preferences#experimental” introduced in Bug 1652134.

Tested Platforms:

  • Windows 10 x64 Intel/Nvidia GPU
  • Windows 7 AMD/Intel GPU
  • macOS 10.15 Nvidia/Intel GPU

Tested Firefox versions:

  • Firefox Nightly 83.0a1 (Build IDs 20201004212809 through 20201009041754)

Regards,
Maria
Ecosystem QA Team

(In reply to Maria Heres, :mheres, Ecosystem QA from comment #3)

Quality status: YELLOW - SHIP IT CONDITIONALLY

Why is this experiment yellow?

  • During testing we found 7 issues, 3 of which are still open. They are not blocking, but one of the open issues has been found late in the testing process and needs the assessment of the Engineering team:

  • Bug 1670263 - The Fission doorhanger is dismissed by some dialogs

Fixing bug 1670263 would be nice (if it doesn't require a lot of custom code to work around the expected doorhanger behavior), but I don't think that bug needs to block launching our Fission experiment.

You need to log in before you can comment on or make changes to this bug.