Closed Bug 1751308 Opened 3 years ago Closed 3 years ago

[Experiment] Pref-Flip Experiment: Tab Unloading on Low Memory for macOS Fx 97.0 to 98.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: experimenter, Assigned: haik)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

browser.lowMemoryResponseMask

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment memory-pressure-warning 20%:

Value: None

Memory-pressure callbacks enabled at the warning level
        
- Treatment tab-unloading-and-memory-pressure-warning 20%:

Value: None

Tab unloading and memory-pressure callbacks at the warning level
        
- Treatment memory-pressure-critical 20%:

Value: None

Memory-pressure callbacks enabled at the critical level
        
- Treatment tab-unloading-and-memory-pressure-critical 20%:

Value: None

Tab unloading and memory-pressure callbacks enabled at the critical level
        
- Control tab-unloading-and-memory-pressure-disabled 20%:

Value: None

No tab unloading and no memory-pressure callbacks.
        

    What version and channel do you intend to ship to?

5% of Release Firefox 97.0 to 98.0

    Are there specific criteria for participants?

None
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Mar 08, 2022 - Apr 05, 2022 (28 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

This is a repeat of the first tab unloading experiment with additional metrics and this time running on Release instead of Nightly.

Original design document:
https://mana.mozilla.org/wiki/display/FIREFOX/Tab+Unloading+Experiment

Background and links to experiments for all platforms:
https://docs.google.com/document/d/1y9eetT3yKZs1TPuDp2BiUfAivGpahywpT5hrQuThWUk/edit#heading=h.3nli4ae604mt

Reference experiment used for list of metrics to measure responsiveness:
https://protosaur.dev/partybal/bug_1722551_pref_full_js_parsing_experiment_nightly_94_94.html

**Metrics List:**

**Tab Unloading Metrics:**

* **Number of tabs unloaded:** We'd like to see how often tab unloading users unload tabs.
* **Number of reloads of unloaded tabs:** We'd like to see how often tab unloading users reload tabs we previously unloaded. A high number means we might be doing a bad job of choosing which tabs to unload.
* **How many times the memory situation was switched from low to high:** Tab unloading users should have more `on_high_memory` events because TabUnloader helps it.
* **tab_reload_cnt**: We'd like to see how often tab unloading users reload tabs we previously unloaded. A high number means we might be doing a bad job of choosing which tabs to unload.
* **tab_unload_to_reload**: We'd like to see how often tab unloading users reload tabs we previously unloaded. A low time means we might be doing a bad job of choosing which tabs to unload.
* **memory\_situation\_switched**

**General Metrics:**

* **active_hours**: Tab unloading users should browse as many or more hours as control users because their browsing session is not cut short by an OOM.
* **active_hrs**
* **ad_clicks**
* **checkerboard_severity**
* **checkerboard\_severity\_count\_per\_hour**
* **child\_process\_launch_ms**
* **content\_crashes\_per\_hour**: Tab unloading users should have fewer content process crashes (because they are a superset of content process OOMs).
* **content\_frame\_time\_vsync**
* **content\_process\_max**
* **cycle\_collecto\r_max\_pause**
* **cycle\_collector\_max\_pause\_content**
* **days\_of\_use**: Tab unloading users should browse as many or more days as control users because they are happier that their Firefox is crashing less.
* **fx\_new\_window\_ms**
* **fx\_tab\_switch\_composite\_e10s\_ms**
* **gc\_budget\_overrun**: 
* **gc\_mark\_rate\_2**: how much did GC slices blow their budget
* **gc\_reason\_2**: frequency & duration of memory pressure GCs (GC_REASON_2 this is an enum, so we want the frequency of 35 over the frequency of all reasons)
* **gc\_max\_pause\_ms\_2**
* **gc\_max\_pause\_ms\_2\_content**
* **gc\_ms**
* **gc\_ms\_content**
* **gc\_non\_incremental**: how often was the GC non-incremental
* **gc\_slice\_during_idle**
* **gc\_slice\_during\_idle\_content**
* **gpu\_keypress\_present\_latency**
* **identity**
* **input\_event\_response\_ms**
* **input\_event\_response\_ms\_parent**
* **js\_pageload\_baseline\_compile\_ms
* **js\_pageload\_delazification\_ms**
* **js\_pageload\_execution\_ms**
* **js\_pageload\_parse\_ms**
* **js\_pageload\_protect\_ms**
* **js\_pageload\_xdr\_encoding\_ms**
* **main\_crashes\_per\_hour**: Tab unloading users should have fewer parent process crashes (because they are a superset of parent process OOMs).
* **max\_concurrent\_tab\_count**
* **max\_concurrent\_tab\_pinned_count**
* **Maximum number of tab opened at the same time:** We'd like to see TabUnloader does not change users tab operation.
* **Maximum number of tab pinned at the same time:** We'd like to see TabUnloader does not change users tab operation.
* **memory\_total: Tab unloading users should use the same or less memory.**
* **memory\_unique\_content\_startup**
* **oom\_crashes\_per\_hour: Tab unloading users should have fewer OOMs.**
* **organic\_search\_count**
* **perf\_first\_contentful\_paint\_ms**
* **perf\_page\_load\_time\_ms**
* **search\_count**
* **searches\_with\_ads**
* **separate\_search\_engine**
* **shutdown\_hangs\_per\_hour**
* **subsession\_length**: Tab unloading users should have longer sessions because their browsing session is not cut short by an OOM.
* **tab\_open\_event\_count**: We'd like to see TabUnloader does not change users tab operation.
* **tab\_pinned\_event\_count**: We'd like to see TabUnloader does not change users tab operation.
* **tagged\_follow\_on\_search\_content**
* **tagged\_follow\_on\_search\_count**
* **tagged\_search\_count**
* **time\_to\_first\_interaction_ms**
* **uri\_cnt**
* **uri\_count**: Tab unloading users should load more URIs because their browsing session is not cut short by an OOM.


Original DO ticket:
https://mozilla-hub.atlassian.net/browse/DO-554

**Estimated Total Enrolled Clients:**
1% of Release Mac users expected to be approximately 80k users.

    Who is the owner of the data analysis for this experiment?

None

    Will this experiment require uplift?

False

    QA Status of your code:

The feature was tested on macOS by QA - https://mozilla-hub.atlassian.net/browse/QA-1196

Experiment-readiness testing has not been done yet.

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/tab-unloading-v2-macos-release/

Tab Unloading V2 - macOS - Release

When Firefox detects the available system memory is becoming low, it will first attempt to reduce its memory usage by freeing memory where possible. If low memory continues, Firefox will then begin unloading the least-recently used tabs. By freeing memory and unloading tabs, we hope to avoid performance problems and crashes caused by low memory.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://mozilla-hub.atlassian.net/browse/DO-554
More information: https://experimenter.services.mozilla.com/experiments/tab-unloading-v2-macos-release/

We've finished running the Experiment Test Cases and sent a GREEN sign-off via email. You can also see it here.

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Tab Unloading V2 - macOS - Release → [Experiment] Pref-Flip Experiment: Tab Unloading on Low Memory for macOS Fx 97.0 to 98.0 Release
Start Date: 2022-03-08 End Date: 2022-04-05
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.