Closed Bug 1694164 Opened 4 years ago Closed 4 years ago

[Experiment] Pref-Flip Experiment: HTTP3 on Beta Fx 87.0 to 89.0 Beta

Tracking

(Not tracked)

Status:

RESOLVED FIXED

People

(Reporter: experimenter, Unassigned)

References

(Blocks 1 open bug,
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

network.http.http3.enabled

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment HTTP3 Enabled 50%:

Value: true

HTTP3 turned on
        
- Control HTTP3 Disables 50%:

Value: false

Default off state for HTTP3
        

    What version and channel do you intend to ship to?

100% of Beta Firefox 87.0 to 89.0

    Are there specific criteria for participants?

Prefs:

Experiments:

Any additional filters:
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Mar 02, 2021 - May 01, 2021 (60 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

We will look at 2 aspects: page load performance and HTTP request error rate.
The page load performance will be analyze by comparing following metrics between the "HTTP3 disabled" group and the "HTTP3 enabled" group:

 - PERF\_PAGE\_LOAD\_TIME\_MS
 - PERF\_FIRST\_CONTENTFUL\_PAINT\_MS

and additionally we will look only at the performance of web sites that support HTTP3:

 - comparing HTTP3\_PERF\_PAGE\_LOAD\_TIME\_MS, both keyed-histograms combined into one, between the "HTTP3 disabled" group and the "HTTP3 enabled" group
 - comparing HTTP3\_PERF\_FIRST\_CONTENTFUL\_PAINT\_MS, both keyed-histograms combined into one, between the "HTTP3 disabled" group and the "HTTP3 enabled" group
 - comparing HTTP3\_PERF\_PAGE\_LOAD\_TIME\_MS histogram with key "http3" of the "HTTP3 enabled" group and the same histogram with key "supports_http3" of the "HTTP3 disabled" group
 - comparing HTTP3\_PERF\_FIRST\_CONTENTFUL\_PAINT\_MS histogram with key "http3" of the "HTTP3 enabled" group and the same histogram with key "supports_http3" of the "HTTP3 disabled" group

The HTTP error rate will be measured by metric HTTP\_CHANNEL\_ONSTART\_SUCCESS. The metrics will be compared between the "HTTP3 disabled" group and the "HTTP3 enabled" group.

Criteria to proceed with turning HTTP3 on by default on Beta are:

 - Performance measurements should be in parity with HTTP2 or better. We expect that PERF\_PAGE\_LOAD\_TIME\_MS and PERF\_FIRST\_CONTENTFUL\_PAINT\_MS will the same or better.
 - Failure rate is expected to be slightly increased with HTTP3. A higher failure rate can be tolerated as long as the performance criteria is satisfied.

More details can be found here: https://docs.google.com/document/d/1EsNj0g08jUrpXNEVf7HvqCiYaEWQq_VzvaboLXzdVJU/

The dashboard can be found here: https://sql.telemetry.mozilla.org/dashboard/http3-preliminary-

    Who is the owner of the data analysis for this experiment?

None

    Will this experiment require uplift?

False

    QA Status of your code:



    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/http3-on-beta/

Experimenter Bot

Reporter

Description

•

4 years ago

HTTP3 on Beta

HTTP3 is a new protocol that will improve the web page load performance.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DS-1455
More information: https://experimenter.services.mozilla.com/experiments/http3-on-beta/

Maria Heres, inactive, contact Ecosystem QA