[Experiment] Pref-Flip Experiment: HTTP3 on Nightly Fx 86.0 to 88.0 Nightly
Categories
(Shield :: Shield Study, task, P3)
Tracking
(firefox86-, firefox87+ disabled, firefox88+ fixed)
People
(Reporter: experimenter, Unassigned)
References
(Blocks 1 open bug, )
Details
(Whiteboard: [no-nag])
User Story
Delivery Type: Pref Flip Experiment
What is the preference we will be changing
None
What are the branches of the experiment and what values should
each branch be set to?
- Treatment http3 enabled treatment 50%:
Value: None
http3 enabled
- Control http3 default off Control 50%:
Value: None
Default off state for http3
What version and channel do you intend to ship to?
100% of Nightly Firefox 86.0 to 88.0
Are there specific criteria for participants?
Prefs:
Experiments:
Any additional filters:
Countries: all
Locales: all
What is your intended go live date and how long will the experiment run?
Jan 26, 2021 - Mar 27, 2021 (60 days)
What is the main effect you are looking for and what data will you use to
make these decisions?
We are planning a rollout to only 50% of the Nightly population and we will observed performance of the feature by comparing the enrolled group and not-enrolled group.
We will look at 2 aspects: the page load performance and the HTTP request error rate.
The page load performance will be analyze by looking at following metrics and comparing them between the enrolled and not-enrolled population:
- PERF\_PAGE\_LOAD\_TIME\_MS
- PERF\_FIRST\_CONTENTFUL\_PAINT\_MS
The HTTP error rate will be measured by metric HTTP\_CHANNEL\_ONSTART\_SUCCESS. The metrics will be compared between the enrolled and not-enrolled group.
Performance measurements should be in parity with the not-enrolled group or better. We expect PERF\_PAGE\_LOAD\_TIME\_MS and PERF\_FIRST\_CONTENTFUL\_PAINT\_MS to be the same or better.
Failure rate is expected to be slightly increased with HTTP3 feature turned on. A higher failure rate can be tolerated as long as the performance criteria is satisfied.
The dashboard can be found here: https://sql.telemetry.mozilla.org/dashboard/http3-preliminary-
Who is the owner of the data analysis for this experiment?
cdowhygelund@mozilla.com
Will this experiment require uplift?
False
QA Status of your code:
Just increase on size from https://experimenter.services.mozilla.com/experiments/http3-performance-study/ - so already passed QA
Link to more information about this experiment:
https://experimenter.services.mozilla.com/experiments/http3-on-nightly/
HTTP3 on Nightly
HTTP3 is a new protocol that will improve the web page load performance.
Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.
Data Science Issue:
More information: https://experimenter.services.mozilla.com/experiments/http3-on-nightly/
|
||
Comment 1•5 years ago
|
||
Turning on HTTP3 on Nightly
Nightly 86, 87, 88
We have finished testing the [PI-923] Turning on HTTP3 on Nightly rollout.
Quality status: GREEN - SHIP IT
Why is this rollout green?
Delivery mechanism:
- No issues found.
Product specific:
- No issues found.
Testing Summary:
Tested Platforms:
- Windows 10 x64
- macOS 10.15
- Ubuntu 20.04 x64
Tested Firefox versions:
- Nightly 86.0a1 en-US (Build ID: 20210119091250)
- Nightly 86.0a1 de (Build ID: 20210119091250)
- Nightly 86.0a1 fr (Build ID: 20210119091250)
|
||
Comment 2•5 years ago
|
||
[Tracking Requested - why for this release]: this is a nightly only experiment. since it is large - we've cleared it with the other large experiment - Fission. Both are fine / could benefit from co-habitation.
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Comment 3•5 years ago
|
||
|
||
Comment 4•5 years ago
|
||
I don't think it needs tracking for 86 as we are already in Beta.
|
||
Updated•5 years ago
|
|
|
||
Updated•5 years ago
|
|
|
||
Comment 5•5 years ago
|
||
87 is now on beta.
|
|
Reporter | |
Updated•5 years ago
|
|
||
Updated•5 years ago
|
Description
•