Closed Bug 1648259 Opened 4 months ago Closed 2 months ago

[Experiment] Pref-Flip Experiment: Performance impact of DLL prefetching Fx 79.0 to 80.0 Beta

Categories

(Shield :: Shield Study, task, P3)

Tracking

(firefox79+ fixed)

RESOLVED FIXED
Tracking Status
firefox79 + fixed

People

(Reporter: experimenter, Assigned: esmyth)

References

(Blocks 1 open bug, )

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

startup.experiments.alteredDllPrefetch

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment Optimized prefetch 33%:

Value: 2

Clients on this branch will use the new optimized prefetch strategy
        
- Treatment No prefetch 33%:

Value: 1

Clients on this branch will not prefetch DLLs
        
- Control Control - Current prefetch 34%:

Value: 0

Clients on this branch will continue using the current prefetch strategy
        

    What version and channel do you intend to ship to?

5% of Beta Firefox 79.0 to 80.0

    Are there specific criteria for participants?

Prefs:

Experiments:

Any additional filters:
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Jul 07, 2020 - Aug 11, 2020 (35 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will you use to make these decisions? What metrics are you using to measure success

Based on what we know,

* Users in the Optimized prefetch should have lower cold startup times than users in the current prefetch
* Users with the current prefetch should have lower cold startup times than the control branch

We want to confirm that the feature does in fact improve these startup times for HDD users, and see whether there is an opposite effect for SSD users. We will use the `timestamps_first_paint_two` metric to test this.

The other main effect we will look for is the impact on retention of this feature.

Do you plan on surveying users at the end of the delivery? No.

    Who is the owner of the data analysis for this experiment?

None

    Will this experiment require uplift?

False

    QA Status of your code:

In progress

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/performance-impact-of-dll-prefetching/

Performance impact of DLL prefetching

We believe that optimizing the DLL pre-fetching can reduce disk IO during startup and thereby decrease time to start Firefox. In local testing on IO-bound machines, we've found the potential for improvement to startup time of up to 25%. This experiment is intended to validate internal test results and guide selection of a pre-fetch strategy.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-267
More information: https://experimenter.services.mozilla.com/experiments/performance-impact-of-dll-prefetching/

This experiment is going out on beta during beta 79 cycle.

Performance impact of DLL prefetching

Firefox Beta 79.0bx

Firefox Beta 80.0bx

We have finished testing the [PI-687] - Performance impact of DLL prefetching experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • No issues were found during testing of the experiment.

Testing Summary:

Tested Platforms:

  • Windows 10 x64
  • Windows 7 x64
  • Windows 8.1 x32

Tested Firefox versions:

  • Firefox Beta 79.0b2 (Build ID: 20200630191632)
  • Firefox Beta 79.0b1 (only for the browser update scenario)
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Performance impact of DLL prefetching → [Experiment] Pref-Flip Experiment: Performance impact of DLL prefetching Fx 79.0 to 80.0 Beta
Start Date: 2020-07-07 End Date: 2020-08-11
Status: NEW → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.