1848459 - Add telemetry to measure how often closed tab data is lost because the window has no open tabs to save

Status: VERIFIED FIXED

(Firefox :: Session Restore, task, P2)

Description

[Sam Foster [:sfoster] (he/him)]

When a window closes, SessionStore implements to checks to determine there is data worth saving in there. If the window has no open tabs (or no tabs that aren't about:home, about:blank etc) we'll skip saving data for that window. This avoids apparently-empty windows showing up in the recently-closed windows menu.

However, if there are useful closed tabs in an otherwise empty window, that data will be lost as result. We'd like to add a telemetry probe to measure how often this happens.

Comment 1

[Sam Foster [:sfoster] (he/him)]

Here's the spot in SessionStore where we are losing that closed tab data. We're handling the closing of a window and trying to figure out if there is useful data we should be storing. hasSaveableTabs only accounts for currently open tabs in that window. The telemetry probe might store the length of winData._closedTabs to know how much if any data we are potentially losing in this state.

Comment 2

[Sarah Clements [:sclements]]

Attachment: Bug 1848459 - add telemetry to count closed tabs lost in session restore r=sfoster

• Add histogram to sessionrestore code to count how often and the number of
  closed tabs that aren't saved on window close due to no open saveable tabs
• Add test coverage

Comment 3

[Sarah Clements [:sclements]]

Comment on attachment 9362564 [details]
Bug 1848459 - add telemetry to count closed tabs lost in session restore r=sfoster

Firefox Data Collection Request - Session Restore

What questions will you answer with this data? It'll tell us how often users close a window that doesn't preserve closed tabs because the only open tab is a new tab. Knowing how often this
happens will help us determine if we need to prioritize bug 1853129.

Why does Mozilla need to answer these questions? Are there benefits for users?
Answering this question will help ensure we have enough information for preserving data that users might expect to be saved from a session.

Can current instrumentation answer these questions? No, because there is no instrumentation for this particular scenerio.

List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Measurement Description Data Collection Category Tracking Bug #
FX_SESSION_RESTORE_CLOSED_TABS_NOT_SAVED 9362564

Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.
This collection is documented in its definitions file Histograms.json and in the Probe Dictionary at https://probes.telemetry.mozilla.org.

FX_SESSION_RESTORE_CLOSED_TABS_NOT_SAVED
Session restore: Number of closed tabs that are NOT saved due to lack of open tabs worth saving on window close.

How long will this data be collected? Choose one of the following:
Until 127, as we only need to see a trend over a 6 month period.

What populations will you measure?
Nightly, Beta, and Release channels, all countries and locales.

If this data collection is default on, what is the opt-out mechanism for users?
Preferences > Privacy & Security > Firefox Data Collection and Use > Deselect the Allow Firefox to send technical and interaction data to Mozilla.

Please provide a general description of how you will analyze this data.
We'll compare the number of closed tabs greater than or equal to 1 to the number of 0 closed tabs in this scenerio to determine if we should prioritize saving those closed tabs ie, assessment is that users are/are not losing a lot of closed tabs that might be valuable, per bug 1853129.

Where do you intend to share the results of your analysis?
Internal Mozilla dashboards and very likely a brief explanation of the data in bug 1853129.

Is there a third-party tool (i.e. not Glean or Telemetry) that you are proposing to use for this data collection? If so:
No

Comment 4

[Sarah Clements [:sclements]]

Whoops, just realized I was supposed to create a new attachment, not add a comment to the phab patch. jhirsch, let me know if you want me to also attach it.... doesn't seem like there's a way to change this.

Comment 5

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Comment on attachment 9362564 [details]
Bug 1848459 - add telemetry to count closed tabs lost in session restore r=sfoster

Sorry for the delay; I've been out sick. (For future reference, if you don't get a data-review within a day, you can ping the data-stewards channel on matrix to see if someone else is available to do the review.)

Whoops, just realized I was supposed to create a new attachment, not add a comment to the phab patch

No worries, we just need to have the data-review attached to the bug in some way or other, so that Firefox users can see for themselves what we are collecting. A comment is fine :-) Note that ./mach data-review (bug number) will automatically generate most of the data review request, including looking at the telemetry yaml files to auto-generate the list of probes; this is helpful when creating a data review for a lot of probes.

data-review+

---

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, the usual Firefox docs.

2. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the usual Firefox preferences to disable data collection.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

N/A - this is a temporary 6 month collection.

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, interaction data.

5. Is the data collection request for default-on or default-off?

Default-on.

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

7. Is the data collection covered by the existing Firefox privacy notice?

Yes.

8. Does the data collection use a third-party collection tool?

No.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0d228f1cfc6f
add telemetry to count closed tabs lost in session restore r=sfoster,sessionstore-reviewers

Comment 7

[Cosmin Sabou [:CosminS]]

Backed out for failures on browser_closed_tabs_closed_windows

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception&fromchange=0d228f1cfc6f794fbf402cb44165116b6255f5c4&test_paths=browser%2Fcomponents%2Fsessionstore%2Ftest&tochange=a0c492fe307073c1550956cb2c0b07e348695c1b&selectedTaskRun=QrpVfoXNSCe4FH74onJ3OA.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=437060866&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/a0c492fe307073c1550956cb2c0b07e348695c1b

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/02c4dd7b226a
add telemetry to count closed tabs lost in session restore r=sfoster,sessionstore-reviewers

Comment 9

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/02c4dd7b226a

Comment 10

[Sarah Clements [:sclements]]

Comment on attachment 9362564 [details]
Bug 1848459 - add telemetry to count closed tabs lost in session restore r=sfoster

Beta/Release Uplift Approval Request

• User impact if declined: It will delay our ability to gather data about whether we should prioritize addressing an edge case: closed tabs that are lost when a window with the only open tabs being new tabs at the time the window is closed.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Open two windows. In the second window open three tabs, two of which should not be new tab. Close those two tabs, leaving the one new tab open. Then close the window. The probe should measure that two closed tabs were lost when the window was closed since it only had one new tab open when it was closed.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It's adding a probe to existing code that has test coverage.
• String changes made/needed: No
• Is Android affected?: No

Comment 11

[Donal Meehan [:dmeehan]]

Comment on attachment 9362564 [details]
Bug 1848459 - add telemetry to count closed tabs lost in session restore r=sfoster

Approved for 121.0b4

Comment 12

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/1a3e4add92e4

Comment 13

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Verified that the telemetry is added in Histograms section and collecting data each time a second window is closed with only a new tab opened. Tests were performed on Firefox 122.0a1 and 121.0b4 on Windows 10, macOS 13.6.1 and Ubuntu 22.04.

Comment 14

[Sam Foster [:sfoster] (he/him)]

This probe was added to assess the number of closed tab entries associated with an otherwise empty window (no non-empty tabs) when it is being closed. In this case we don't create a entry for the window in the session state - meaning those closed tabs are lost. Its an edge case and we wanted to understand what impact this is having in the wild.

Turns out, of the people who hit this probe (i.e. are closing an otherwise empty window), around 70-80% have 0 closed tabs, ~6% have one closed tab and the curve for 2 closed tabs, 3, etc. goes down in the expected pattern. 0.5% have 12 closed tab entries - which was the largest number in the sample I looked at.

I think that is roughly in line with what we suspected. We haven't had bugs filed about loss of data for this case. But, I'm not sure we should read too much into that. It turns out there is some confusion and ambiguity around when the user is ending a session and what data is going to be captured for restore next session, and when exactly that occurs. So any messaging or mitigation we consider should bear that in mind.