Closed Bug 1210773 Opened 9 years ago Closed 9 years ago

Add telemetry to analyze usage of tab groups / panorama

Categories

(Firefox Graveyard :: Panorama, defect)

defect
Not set
normal

Tracking

(firefox44 affected)

RESOLVED WORKSFORME
Tracking Status
firefox44 --- affected

People

(Reporter: Gijs, Assigned: Gijs)

Details

In order to make a decisions about panorama and its replacements/future/lack-thereof, it would be useful if we had actual data about its use.
Seems like we can get some basic data by checking:

- the number of tab groups along with the number of open tabs in BrowserUITelemetry (so as part of simple measurements)
- the number of times panorama/tabview is opened as a countable event in there.
Err, in fact it seems there was telemetry for this, but it expired in version 40. See e.g. 

https://telemetry.mozilla.org/advanced/#filter=nightly%2F39%2FPANORAMA_GROUPS_COUNT%2Fsaved_session%2FFirefox&aggregates=Submissions&evoOver=Builds&locked=true&sanitize=true&renderhistogram=Graph

For the number of submissions.

It looks like the number of submissions of that thing is literally dwarfed by any other measure I can find, like say, the number of people who have the XUL cache disabled:

https://telemetry.mozilla.org/advanced/#filter=nightly%2F39%2FSTARTUP_XUL_CACHE_DISABLED%2Fsaved_session%2FFirefox&aggregates=Submissions&evoOver=Builds&locked=true&sanitize=true&renderhistogram=Graph

Dolske, how do you want me to proceed here?
Flags: needinfo?(dolske)
OTOH... unless I misunderstand the code here ( https://dxr.mozilla.org/mozilla-central/source/browser/components/tabview/telemetry.js , included from https://dxr.mozilla.org/mozilla-central/source/browser/components/tabview/tabview.js, included from the HTML file - init/uninit being called from https://dxr.mozilla.org/mozilla-central/source/browser/components/tabview/ui.js ) it seems like this telemetry will only be included if gather-telemetry (idle-daily) fires while tabview is up/open. That seems... wrong? Maybe I do misunderstand the code? Seems like it was written in bug 666538, Tim, can you clarify?
Flags: needinfo?(ttaubert)
(In reply to :Gijs Kruitbosch from comment #3)
> it seems like this telemetry will only be included if gather-telemetry
> (idle-daily) fires while tabview is up/open. That seems... wrong? Maybe I do
> misunderstand the code?

tabview.html stays loaded in TabView._iframe after init even after closing the view so the user doesn't have to be viewing the groups but had to init it once. This means it won't count people who didn't open panorama in that session so it will under count.
(In reply to Matthew N. [:MattN] (PTO Oct. 8-9) from comment #4)
> (In reply to :Gijs Kruitbosch from comment #3)
> > it seems like this telemetry will only be included if gather-telemetry
> > (idle-daily) fires while tabview is up/open. That seems... wrong? Maybe I do
> > misunderstand the code?
> 
> tabview.html stays loaded in TabView._iframe after init even after closing
> the view so the user doesn't have to be viewing the groups but had to init
> it once. This means it won't count people who didn't open panorama in that
> session so it will under count.

Thanks for clarifying. Would still like to know if we need more data here and/or should adjust how we collect data.
Flags: needinfo?(ttaubert)
Looking at Beta 31-39 data from http://mzl.la/1NqqFks:

The rough average seems to be 1500-2000 submissions/day, and (from a different dashboard) we had ~8M "Unique Active Instances" in that timeframe. So that puts usage in the 0.025% ballpark.

(Although, curiously, that started dropping in 38, and the 39 data shows ~500/day. I wonder why. Real, or data glitch?)


Looking at Aurora 31-39 data from http://mzl.la/1NqqJRh:

The rough average seems to be 150 submissions/day before Nov 2014 (when Developer edition was released), and 250/day afterwards. ~160K and ~250K UAI for the two time ranges, and both work out to usage in the 0.01% ballpark.

(The graphs show a weird, 7x spiked datapoint at the end of each release cycle, but this shouldn't meaningfully affect the numbers.)


That's basically non-existent usage, which confirms our suspicion that it was extremely low. (Data for Release channel would be nice, but since we actually already had a probe with a long history for Aurora/Beta, I don't think that's needed now.) In fairness, these are not super-precise numbers: I'm mixing ADIs with telemetry submission counts and (as previous comments note) we only record these probes if the Tab View UI was opened in the session. But for any reasonable multiplier to adjust for that, usage is still _extremely_ low.

So I think it's clear should remove this feature, and we just need to decide on a migration strategy.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Flags: needinfo?(dolske)
Resolution: --- → WORKSFORME
Hi Justin,

Commenting as a concerned current user of Panorama who would prefer that it not be removed. Can you explain why you used Beta/Aurora data for your calculations above and not Release data? I think almost everyone I know uses this Firefox feature, and when I look at the release data vs. the beta/alpha data, there are thousands more users; release data dwarfs the beta data. (Of course, I don't know the denominators you're using for calculating the percentages, so maybe the differences aren't that significant.)

Thanks,

- e
Yes, the pre-release audiences are smaller than Release. But the numbers are percentages of the respective channel, so that doesn't really matter. And from previous experience I'd expect that the Release percentages would be even lower (generally, pre-release audiences tend to be early-adopters / more technically inclined, and thus use more advances/complex features more).

I'd have used Release data if we had it, but most data collection is disabled on Release for privacy reasons. Given the above, and that the existing probe had already been collecting pre-release for a long period of time that shows _extremely_ low usage, no further data was needed.
Product: Firefox → Firefox Graveyard
You need to log in before you can comment on or make changes to this bug.