Closed Bug 1751309 Opened 3 years ago Closed 3 years ago

[Experiment] Pref-Flip Experiment: Tab Unloading on Low Memory for Windows Fx 97.0 to 98.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: experimenter, Assigned: haik)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

browser.tabs.unloadOnLowMemory

    What are the branches of the experiment and what values should
    each branch be set to?

- Control tab-unloading-disabled 50%:

Value: false

Tab Unloading is disabled.
        
- Treatment tab-unloading-enabled 50%:

Value: true

Tab Unloading is enabled.
        

    What version and channel do you intend to ship to?

1% of Release Firefox 97.0 to 98.0

    Are there specific criteria for participants?

None
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Mar 08, 2022 - Apr 05, 2022 (28 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

This is a repeat of the first tab unloading experiment with additional metrics and this time running on Release instead of Nightly.

Original design document:
https://mana.mozilla.org/wiki/display/FIREFOX/Tab+Unloading+Experiment

Background and links to experiments for all platforms:
https://docs.google.com/document/d/1y9eetT3yKZs1TPuDp2BiUfAivGpahywpT5hrQuThWUk/edit#heading=h.3nli4ae604mt

Reference experiment used for list of metrics to measure responsiveness:
https://protosaur.dev/partybal/bug1722551preffulljsparsingexperimentnightly94_94.html

**Metrics List:**

**Tab Unloading Metrics:**

* **Number of tabs unloaded:** We'd like to see how often tab unloading users unload tabs.
* **Number of reloads of unloaded tabs:** We'd like to see how often tab unloading users reload tabs we previously unloaded. A high number means we might be doing a bad job of choosing which tabs to unload.
* **How many times the memory situation was switched from low to high:** Tab unloading users should have more `on_high_memory` events because TabUnloader helps it.
* **tab_reload_cnt**: We'd like to see how often tab unloading users reload tabs we previously unloaded. A high number means we might be doing a bad job of choosing which tabs to unload.
* **tab_unload_to_reload**: We'd like to see how often tab unloading users reload tabs we previously unloaded. A low time means we might be doing a bad job of choosing which tabs to unload.
* **memory\_situation\_switched**

**General Metrics:**

* **active_hours**: Tab unloading users should browse as many or more hours as control users because their browsing session is not cut short by an OOM.
* **active_hrs**
* **ad_clicks**
* **checkerboard_severity**
* **checkerboard\_severity\_count\_per\_hour**
* **child\_process\_launch_ms**
* **content\_crashes\_per\_hour**: Tab unloading users should have fewer content process crashes (because they are a superset of content process OOMs).
* **content\_frame\_time\_vsync**
* **content\_process\_max**
* **cycle\_collecto\r_max\_pause**
* **cycle\_collector\_max\_pause\_content**
* **days\_of\_use**: Tab unloading users should browse as many or more days as control users because they are happier that their Firefox is crashing less.
* **fx\_new\_window\_ms**
* **fx\_tab\_switch\_composite\_e10s\_ms**
* **gc\_budget\_overrun**: 
* **gc\_mark\_rate\_2**: how much did GC slices blow their budget
* **gc\_reason\_2**: frequency & duration of memory pressure GCs (GC_REASON_2 this is an enum, so we want the frequency of 35 over the frequency of all reasons)
* **gc\_max\_pause\_ms\_2**
* **gc\_max\_pause\_ms\_2\_content**
* **gc\_ms**
* **gc\_ms\_content**
* **gc\_non\_incremental**: how often was the GC non-incremental
* **gc\_slice\_during_idle**
* **gc\_slice\_during\_idle\_content**
* **gpu\_keypress\_present\_latency**
* **identity**
* **input\_event\_response\_ms**
* **input\_event\_response\_ms\_parent**
* **js\_pageload\_baseline\_compile\_ms
* **js\_pageload\_delazification\_ms**
* **js\_pageload\_execution\_ms**
* **js\_pageload\_parse\_ms**
* **js\_pageload\_protect\_ms**
* **js\_pageload\_xdr\_encoding\_ms**
* **main\_crashes\_per\_hour**: Tab unloading users should have fewer parent process crashes (because they are a superset of parent process OOMs).
* **max\_concurrent\_tab\_count**
* **max\_concurrent\_tab\_pinned_count**
* **Maximum number of tab opened at the same time:** We'd like to see TabUnloader does not change users tab operation.
* **Maximum number of tab pinned at the same time:** We'd like to see TabUnloader does not change users tab operation.
* **memory\_total: Tab unloading users should use the same or less memory.**
* **memory\_unique\_content\_startup**
* **oom\_crashes\_per\_hour: Tab unloading users should have fewer OOMs.**
* **organic\_search\_count**
* **perf\_first\_contentful\_paint\_ms**
* **perf\_page\_load\_time\_ms**
* **search\_count**
* **searches\_with\_ads**
* **separate\_search\_engine**
* **shutdown\_hangs\_per\_hour**
* **subsession\_length**: Tab unloading users should have longer sessions because their browsing session is not cut short by an OOM.
* **tab\_open\_event\_count**: We'd like to see TabUnloader does not change users tab operation.
* **tab\_pinned\_event\_count**: We'd like to see TabUnloader does not change users tab operation.
* **tagged\_follow\_on\_search\_content**
* **tagged\_follow\_on\_search\_count**
* **tagged\_search\_count**
* **time\_to\_first\_interaction_ms**
* **uri\_cnt**
* **uri\_count**: Tab unloading users should load more URIs because their browsing session is not cut short by an OOM.


Original DO ticket:
https://mozilla-hub.atlassian.net/browse/DO-554

**Estimated Total Enrolled Clients:**

1. Release Windows WAUs = 

2. XX control users + XX treatment users

    Who is the owner of the data analysis for this experiment?

None

    Will this experiment require uplift?

False

    QA Status of your code:

QA did experiment readiness testing of this feature for the earlier experiment. The feature has already shipped to Release and doesn't need further QA testing.

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/tab-unloading-v2-windows-release/

Tab Unloading V2 - Windows - Release

When Firefox detects the available system memory is becoming low, it will first attempt to reduce its memory usage by freeing memory where possible. If low memory continues, Firefox will then begin unloading the least-recently used tabs. By freeing memory and unloading tabs, we hope to avoid performance problems and crashes caused by low memory.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://mozilla-hub.atlassian.net/browse/DO-554
More information: https://experimenter.services.mozilla.com/experiments/tab-unloading-v2-windows-release/

We've finished running the Experiment Test Cases and sent a GREEN sign-off via email. You can also see it here.

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Tab Unloading V2 - Windows - Release → [Experiment] Pref-Flip Experiment: Tab Unloading on Low Memory for Windows Fx 97.0 to 98.0 Release
Start Date: 2022-03-08 End Date: 2022-04-05
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.