[Experiment] Pref-Flip Experiment: Ongoing Fission Nightly experiment Fx 83.0 to 100.0 Nightly
Categories
(Shield :: Shield Study, task, P3)
Tracking
(firefox81 wontfix, firefox82 wontfix, firefox83 affected)
People
(Reporter: experimenter, Assigned: cpeterson)
References
(Blocks 1 open bug, )
Details
(Whiteboard: [fission:m6b])
User Story
Delivery Type: Pref Flip Experiment
What is the preference we will be changing
fission.experiment.enrollmentStatus
What are the branches of the experiment and what values should
each branch be set to?
- Treatment fission-enabled 50%:
Value: 2
This branch has Fission enabled.
- Control fission-disabled 50%:
Value: 1
The control group does not have Fission enabled.
What version and channel do you intend to ship to?
5% of Nightly Firefox 83.0 to 100.0
Are there specific criteria for participants?
Filters:
Exclude users with Fission already configured (user branch value): `fission.autostart` pref has USER value. We don't want users who are already dogfooding Fission to be part of this experiment.
- Require WebRender for Fission on Windows and macOS for our Nightly experiment. Users must be WebRender "qualified": `normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == ‘available’`. (We hope to eventually ship support for Fission with or without WebRender, but we will require it now to simplify our Nightly experiment.)
- Exclude users in any WebRender experiments. The current WebRender v2 experiment (https://experimenter.services.mozilla.com/experiments/webrender-continued-v2-nightly-only/) slug is “bug-1622934-pref-webrender-continued-v2-nightly-only-nightly-76-80”.
The Fission Experiment Plan (https://docs.google.com/document/d/1ZXAHep4v1lW-ZpLMtnTXAtbXSNP8Q-gr-RjUha2Yb14/edit#heading=h.dj0tcj3def9) has more detailed explanations of these eligibility criteria.
Corey's Colab notebook estimating we will need 5% sample size to get 1000 users/branch: https://sql.telemetry.mozilla.org/queries/74160/source
Countries: all
Locales: all
What is your intended go live date and how long will the experiment run?
Oct 06, 2020 - Oct 06, 2021 (365 days)
What is the main effect you are looking for and what data will you use to
make these decisions?
What is the main effect you are looking for and what data will you use to make these decisions? What metrics are you using to measure success?
We want to be able to monitor the difference in stability and performance metrics between clients with and without Fission enabled. The [Fission Experiment Plan](https://docs.google.com/document/d/1ZXAHep4v1lW-ZpLMtnTXAtbXSNP8Q-gr-RjUha2Yb14/edit#heading=h.asctkf7cbxa7) has a detailed list of telemetry probes (and expectations for each probe: up, down, or flat).
We plan to monitor these existing metrics:
- `CHECKERBOARDING_SEVERITY`
- `CHILD_PROCESS_LAUNCH_MS`
- `CONTENT_FRAME_TIME_VSYNC`
- `SCALARS_GFX.OMTP.PAINT_WAIT_RATIO`
- `FX_NEW_WINDOW_MS`
- `FX_TAB_SWITCH_COMPOSITE_E10S_MS`
- `KEYPRESS_PRESENT_LATENCY_MS`
- `INPUT_EVENT_RESPONSE_MS`
- `MEMORY_TOTAL`
- `CYCLE_COLLECTOR_MAX_PAUSE`
- `GC_MAX_PAUSE_MS_2`
- `GC_MS`
- `GC_SLICE_DURING_IDLE`
- `MEMORY_UNIQUE_CONTENT_STARTUP`
- `FX_NUMBER_OF_UNIQUE_SITE_ORIGINS_PER_LOADED_TABS`
- `FX_NUMBER_OF_UNIQUE_SITE_ORIGINS_ALL_TABS`
- `TIME_TO_FIRST_CONTENTFUL_PAINT_MS`
- `TIME_TO_FIRST_INTERACTION_MS`
- `FX_PAGE_LOAD_MS_2`
- `LOADED_TAB_COUNT`
- `browser.engagement.total_uri_count`
- `browser.engagement.tab_open_event_count`
- `browser.engagement.max_concurrent_tab_count`
- Number of users submitting pings
- Main process crash rate (per 1000 usage hours), 32-bit and 64-bit
- Content process crashes (per 1000 usage hours) (excluding ShutdownKills), 32-bit and 64-bit
- `oom_crashes` (per 1000 usage hours), 32-bit and 64-bit
- `shutdown_crashes` (per 1000 usage hours)
- GPU process crashes (per 1000 usage hours)
Do you plan on surveying users at the end of the delivery? No.
Estimated Total Enrolled Clients: 2500 (assuming we enroll 5% of ~50,000 Nightly users)
Who is the owner of the data analysis for this experiment?
cdowhygelund@mozilla.com
Will this experiment require uplift?
False
QA Status of your code:
Not started
Link to more information about this experiment:
https://experimenter.services.mozilla.com/experiments/fission-nightly/
Ongoing Fission Nightly experiment
Track the stability and performance effects of Fission
Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.
Data Science Issue: https://jira.mozilla.com/browse/DS-753
More information: https://experimenter.services.mozilla.com/experiments/fission-nightly/
|
Assignee | |
Updated•4 years ago
|
|
|
Assignee | |
Comment 1•4 years ago
|
||
We are aiming to launch our Fission Nightly experiment on Oct 6 (Fx83 Nightly).
|
Reporter | |
Updated•4 years ago
|
|
|
Assignee | |
Updated•4 years ago
|
|
|
Reporter | |
Comment 2•4 years ago
|
||
|
||
Comment 3•4 years ago
|
||
Ongoing Fission Nightly experiment
Nightly Firefox 83.0 to 100.0
We have finished testing the Ongoing Fission Nightly experiment experiment.
Quality status: YELLOW - SHIP IT CONDITIONALLY
Why is this experiment yellow?
-
During testing we found 7 issues, 3 of which are still open. They are not blocking, but one of the open issues has been found late in the testing process and needs the assessment of the Engineering team:
-
Bug 1670263 - The Fission doorhanger is dismissed by some dialogs
-
An enhancement has recently been opened, Bug 1670587, that still needs to land and to be verified by QA.
What needs to be done?
-
[Engineering] The team needs to assess whether Bug 1670263 needs to be fixed before launch of the experiment based on how often it is likely to be encountered.
-
[QA] We will need to verify Bug 1670263 in the case it is fixed.
-
[QA] We will verify Bug 1670587 when it has landed.
Testing Summary:
-
Test runs: TestRail.
-
Exploratory testing related to:
-
enrollment and unenrollment;
-
the doorhanger;
-
the interaction between the experiment and the addition of Fission to the list of experimental features found in “about:preferences#experimental” introduced in Bug 1652134.
Tested Platforms:
- Windows 10 x64 Intel/Nvidia GPU
- Windows 7 AMD/Intel GPU
- macOS 10.15 Nvidia/Intel GPU
Tested Firefox versions:
- Firefox Nightly 83.0a1 (Build IDs 20201004212809 through 20201009041754)
Regards,
Maria
Ecosystem QA Team
|
|
Assignee | |
Comment 4•4 years ago
|
||
(In reply to Maria Heres, :mheres, Ecosystem QA from comment #3)
Quality status: YELLOW - SHIP IT CONDITIONALLY
Why is this experiment yellow?
During testing we found 7 issues, 3 of which are still open. They are not blocking, but one of the open issues has been found late in the testing process and needs the assessment of the Engineering team:
Bug 1670263 - The Fission doorhanger is dismissed by some dialogs
Fixing bug 1670263 would be nice (if it doesn't require a lot of custom code to work around the expected doorhanger behavior), but I don't think that bug needs to block launching our Fission experiment.
|
||
Comment 5•4 years ago
|
||
Fission nightly experiment has been running since Fx83. Nothing more to do here.
|
|
||
Updated•4 years ago
|
|
|
Reporter | |
Updated•4 years ago
|
Description
•