1602903 - Collect telemetry on the number of Fission versus non-Fission windows

Status: RESOLVED FIXED

(Core :: DOM: Navigation, enhancement, P1)

Description

[Corey Dow-Hygelund [:ccd]]

Attachment: fission_nonfission_window_count.txt

The proposal is to collect the number of Fission versus non-Fission content processes. This will take a snapshot of a minimum 5 minutes interval, when the browser isn't idle.

Comment 1

[Chris Peterson [:cpeterson]]

Tracking for Fission dogfooding (M5)

Comment 2

[Neha Kochar [:neha]]

Sean said he'll work on adding this.

Comment 3

[Olli Pettay [:smaug][[e-mail]]]

I don't know what "Fission and non-Fission windows" means, but P2, given that this is assigned.
neha, is there perhaps some clarification what this bug is about? (just thinking whoever might review this needs to know what the behavior should be)

Comment 4

[Chris Peterson [:cpeterson]]

(In reply to Olli Pettay [:smaug] from comment #3)

I don't know what "Fission and non-Fission windows" means

Nightly now has a "File > New Non-Fission Window" command, so one browsing session can have multiple Fission and non-Fission windows. The Data Science team worries that telemetry from non-Fission windows will pollute the Fission telemetry results from sessions that have both window types open. We want to measure how often that happens.

@ Corey, are you suggesting we report the cumulative number of Fission and non-Fission windows? Or just the numbers in mixed Fission/non-Fission sessions? Should we report the number of Fission and non-Fission windows for sessions that are only Fission or only non-Fission?

Comment 5

[Chris Peterson [:cpeterson]]

Corey, does your suggestion (in this morning's Fission Metrics meeting) that we tag Fission and non-Fission telemetry results using keyed histograms make this Fission/non-Fission window count probe unnecessary?

Comment 6

[Corey Dow-Hygelund [:ccd]]

This I believe still has utility, to help investigate cases where we can't use keyed histograms to split out Fission and non-Fission(e.g., Cycle collection).

This collection mechanism of this probe is identical to the site-origin probe. This should be collected in all cases, including scenarios where only Fission or non-Fission windows were used.

Comment 7

[Sean Feng [:sefeng]]

Olli, do you think the above explanations valid?

Comment 8

[Olli Pettay [:smaug][[e-mail]]]

I don't know. I still don't quite understand what is being measured. How would this help with cycle collection probes (which comment 6 is talking about)?
Or if this is just about Fission vs. non-Fission top level windows, how would we use that data?
I feel like I'm missing something here.

Comment 9

[Sean Feng [:sefeng]]

I see, yeah that's fair.

Corey, could you please clarify how we would we use the data?

Comment 10

[Corey Dow-Hygelund [:ccd]]

It would enable us to segment the data into cases where Fission was only used, Fission+non-Fission was used, and only non-Fission was used. This will help analyzes regarding Fission's impact on certain metrics, such as the cycle collection probes, where we can't split out the metric into Fission and non-Fission cases. As an example, we could compare the cycle collection metrics for cases where only Fission was used, versus cases where Fission was never used.

It will also give us an idea as to how often and to what degree dual windows (sessions with Fission and non-Fission windows) are used in the wild.

Comment 11

[Sean Feng [:sefeng]]

Alright, after speaking with Corey, I think I get the idea.

Instead of using histograms, we'll be using keyed scalar, basically the data we reported will be like {'fission_windows': <count>, 'non_fission_windows': <count>} for each browsing sessions.

Once we have the data, we can write some queries to get a sense of Fission vs non Fission windows (as Corey mentioned in Comment 10).

Since the keyed scalar and the cycle collection histogram data are part of the main ping, we can use Bigquery to write some queries to learn how cycle collection performs in different fission window counts.

Does it clear, Olli?

Comment 12

[Olli Pettay [:smaug][[e-mail]]]

Ok, that is fine.
(Though I do doubt a bit that anyone dealing with cycle collector for example will run any special queries to see how CC behaves.
But perhaps telemetry UI will have something to check how things behave with or without Fission - that would be handy. )

Comment 13

[Chris Peterson [:cpeterson]]

Raising priority to P1 because we should prioritize adding this Fission probe over the other Fission-specific perf probes. Those probes would only be necessary if we find that quite a few users are running both Fission and non-Fission windows in the same browsing session. If that is common scenario, the non-Fission windows' telemetry would pollute the Fission telemetry. If that is a rare scenario, then we probably don't need to bother implementing all those Fission-specific perf probes.

Unassigning Sean for now because the Perf team is focusing on Fenix in Q1. Hopefully someone on the DOM Fission team will have time to implement this probe in Q1.

Comment 14

[Nika Layzell [:nika] (ni? for response)]

A few questions about what data we want to be collecting here:
With this probe, do we want to count fission vs. non-fission tabs, or fission vs. non-fission toplevel windows?
Do we want the count to accumulate across an entire session?

Comment 15

[David Teller [:Yoric] - still alive but not very active]

I'm available to work on this once comment 14 has been answered.

Comment 16

[Corey Dow-Hygelund [:ccd]]

With this probe, do we want to count fission vs. non-fission tabs, or fission vs. non-fission toplevel windows?

toplevel windows

Do we want the count to accumulate across an entire session?

Yes

Comment 17

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1602903 - Collect per-window useRemoteSubframes telemetry,

This telemetry records, for each toplevel window, the relationship between the
actual fission-enabled status of the window, and the "fission.autostart" pref.

This takes the form of a categorized telelemtry histogram with the following
keys:

• EnabledByAutostart: useRemoteSubframes == true fission.autostart == true
• EnabledByUser: useRemoteSubframes == true fission.autostart == false
• DisabledByAutostart: useRemoteSubframes == false fission.autostart == false
• DisabledByUser: useRemoteSubframes == false fission.autostart == true

'DisabledByUser' helps indicate useful information such as how often fission
testers need to open non-fission windows due to breakage, while
'EnabledByAutostart' and 'EnabledByUser' helps give a better image of how many
people are testing fission.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8c683ce02fba
Collect per-window useRemoteSubframes telemetry, r=mconley

Comment 19

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/8c683ce02fba