1181500 - Instrumentation for tablet mode usage

Status: VERIFIED FIXED

(Firefox :: General, defect, P1)

Description

[(Currently slow to respond) Philipp Sackl [:phlsa] (Firefox UX) please use needinfo]

We have a bunch of tablet mode related issues in Firefox.
In order to have some sense of priority and severity of those bugs (as well as for prioritizing tablet mode specific new enhancements), we need some knowledge on how tablet mode is being used.

Here's the kinds of questions I think we need to get answered. Bryan, Dolske, feel free to edit.

- What % of sessions were fully or partially in tablet mode?
- How long are sessions in tablet mode (vs. normal session length)
- How often does the same user use tablet mode with Firefox? Another way of asking that is: do users continue using Firefox and tablet mode together after discovering it?

Since I assume that telemetry data would be too biased to be of any use here, this would probably need to be done in FHR.

Comment 1

[:Gijs (he/him)]

What's the process for getting this privacy-reviewed/approved/whatever, and are there examples of how to add this kind of data in FHR?

(also useful: how easy is implementing this on trunk and uplifting to 40, because FHR is going away but isn't dead in 40 yet?)

Comment 2

[Benjamin Smedberg]

The generic process is documented at https://wiki.mozilla.org/Firefox/Data_Collection

Describe the data you want to collect in the form of a patch to the in-tree docs, and I or a peer can review them. Do you have docs that describe what tablet mode is? Is it a mode of Windows, or a mode of Firefox itself? Can a Firefox session switch between the two modes, and how common is that likely to be?

FHR is not a good match for the questions you're asking, because it doesn't typically collect session information (it aggregates by UTC day).

I don't think you're going to get this into Firefox 40 either way, so I recommend collecting this in unified telemetry, and perhaps adding it as part of the environment which will give us accurate correlations.

Comment 3

[Jared Wein [:jaws] (please needinfo? me)]

Attachment: Patch

Looking at WindowsUIUtils.inTabletMode, it will return early if not on Windows 10 or later, so the inTabletMode value will always return `false` on non-Win10.

I didn't see a better group within TelemetryEnvironment so I put the data within the OS group. Mobile devices have `isTablet` in their DeviceData section.

Comment 4

[:Gijs (he/him)]

Comment on attachment 8650008 [details] [diff] [review]
Patch

For what it does, this code is fine.

However, I worry that this doesn't seem like it'll answer the questions in comment #0. This will just indicate whether we happened to be in or out of tablet mode at the time of the EnvironmentCache construction. We'll have no idea of whether users go into / out of tablet mode while Firefox is open, or if they continue using it like this (which is a hard question to answer anyway, because what constitutes "using" - does leaving it open in the background while doing something else count?)

Comment 5

[Jared Wein [:jaws] (please needinfo? me)]

Attachment: Patch for documentation update

> Do you have docs that describe what tablet mode is? Is it a mode of Windows, or a mode of Firefox itself?

Tablet mode is a mode of Windows 10, which shows applications in fullscreen similar to Windows 8 Metro. Firefox may alter its behavior slightly when it finds that Tablet Mode is enabled.

> Can a Firefox session switch between the two modes, and how common is that likely to be?

Windows can switch between Tablet Mode enabled and disabled without restart. Some changes can be made without user interaction such as plugging in an external monitor (automatic disabling) or manually enabling/disabling via the Windows 10 Notification Area toggle.

We don't have data yet to know how common it is to switch between modes.

Comment 6

[Benjamin Smedberg]

Comment on attachment 8650564 [details] [diff] [review]
Patch for documentation update

I think it's perfectly find to record whether we're in tablet mode or not in telemetry.

It's not clear to me whether we should record this in the environment. If we do record this in the environment, we must keep it live up-to-date: a change to tablet-mode should cause a subsession split.

The advantage of causing a subsession split is that if you want to correlate other metrics, you can do so accurately. The major disadvantage is that subsession splits can be expensive and so we want to minimize them.

For now I'd recommend recording this as a boolean histogram named WINDOWS_IS_TABLETMODE: at startup, record a single value, and then if it changes record additional values. This won't get you correlations, but it should allow you to answer all of the questions from comment 0.

Comment 7

[Benjamin Smedberg]

Comment on attachment 8650564 [details] [diff] [review]
Patch for documentation update

This is r+ because if you really do want to add this to the environment and do the splitting work, that's ok. But it seems like overkill and I'd recommend against it.

Comment 8

[Jared Wein [:jaws] (please needinfo? me)]

Attachment: Patch v2

This patch uses the "tablet-mode-change" observer notification and counts the number of times that tablet mode is entered (including if Firefox is started in tablet-mode) during a session.

We would like to monitor this data to understand how tablet mode is being used and to provide enough resources to support its usage. Users in tablet mode likely have different usage behaviors, such as potentially needing an on-screen keyboard and larger buttons, both of which have poor support for today.

Comment 9

[Tim Taubert [:ttaubert] (inactive)]

Comment on attachment 8656144 [details] [diff] [review]
Patch v2

Review of attachment 8656144 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM.

Comment 10

[Allison Naaktgeboren :ally]

Comment on attachment 8656144 [details] [diff] [review]
Patch v2

Review of attachment 8656144 [details] [diff] [review]:
-----------------------------------------------------------------

So the code as written counts the number of times we enter tablet mode. It does not include the session splitting that bsmedberg has mentioned, so you will not be able to draw correlations, including determining how long a user spent in tablet mode.  As long as you're cool with that, I'm cool with that.

To bring it back around to the original questions, I think you'll get enough data to start making some decisions.
Will this probe answer this question 
- What % of sessions were fully or partially in tablet mode? Yes
- How long are sessions in tablet mode (vs. normal session length) No
- How often does the same user use tablet mode with Firefox? Not solidly, but you'll get a pretty good idea. Telemetry is not very longitudinal.  

p=ally

Comment 11

[Jared Wein [:jaws] (please needinfo? me)]

(In reply to Allison Naaktgeboren please NEEDINFO? :ally from comment #10)
> Comment on attachment 8656144 [details] [diff] [review]
> Patch v2
> 
> Review of attachment 8656144 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> So the code as written counts the number of times we enter tablet mode. It
> does not include the session splitting that bsmedberg has mentioned, so you
> will not be able to draw correlations, including determining how long a user
> spent in tablet mode.  As long as you're cool with that, I'm cool with that.
> 
> To bring it back around to the original questions, I think you'll get enough
> data to start making some decisions.
> Will this probe answer this question 
> - What % of sessions were fully or partially in tablet mode? Yes
> - How long are sessions in tablet mode (vs. normal session length) No
> - How often does the same user use tablet mode with Firefox? Not solidly,
> but you'll get a pretty good idea. Telemetry is not very longitudinal.  
> 
> p=ally

Thanks, yes, this will be sufficient for now. We can iterate on this in the future if we feel we need more fine-grained data.

Comment 12

[Jared Wein [:jaws] (please needinfo? me)]

url:        https://hg.mozilla.org/integration/fx-team/rev/09ad75a2d51f2e8154c88da3b4d8a1951aa5f77a
changeset:  09ad75a2d51f2e8154c88da3b4d8a1951aa5f77a
user:       Jared Wein <[e-mail]>
date:       Thu Sep 03 14:34:29 2015 -0400
description:
Bug 1181500 - Instrumentation for tablet mode usage in desktop Firefox. r=ttaubert p=ally

Comment 13

[Sheila Mooney]

[Tracking Requested - why for this release]: This is close. Adding the tracking flag for 41 per Ritu's request.

Comment 14

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/09ad75a2d51f

Comment 15

[Jared Wein [:jaws] (please needinfo? me)]

Attachment: Patch for Aurora42

Approval Request Comment
[Feature/regressing bug #]: telemetry tracking for windows 10 features
[User impact if declined]: delayed data collection
[Describe test coverage new/current, TreeHerder]: manual verification of telemetry results
[Risks and why]: low risk, telemetry only
[String/UUID change made/needed]: none

Comment 16

[Jared Wein [:jaws] (please needinfo? me)]

Attachment: Patch for Beta41

Approval Request Comment
[Feature/regressing bug #]: telemetry tracking for windows 10 features
[User impact if declined]: delayed data collection
[Describe test coverage new/current, TreeHerder]: manual verification of telemetry results
[Risks and why]: low risk, telemetry only
[String/UUID change made/needed]: none

Comment 17

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8657153 [details] [diff] [review]
Patch for Aurora42

Additional telemetry data around tablet mode use is good and will help us to make the right data driven decisions. Aurora42+

Comment 18

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8657154 [details] [diff] [review]
Patch for Beta41

Unified Telemetry is pref'd off by default in FF41. The risk associated with this uplift is therefore pretty low. Beta41+

Comment 19

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

I wanted to track this in case the fix did not land soon enough so it would stay on my radar. Given that the fix has landed, I do not feel the need to track this. Please let me know if there are any concerns.

Comment 20

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/c60e76aba39d

Comment 21

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/releases/mozilla-beta/rev/2aba90d02099

Comment 22

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/d5686c4eb3e5

Comment 23

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I accidentally landed the m-c version of the patch to aurora (didn't catch that there was an aurora-specific patch posted until after the fact). Backed out the original and landed the new one in ^

Comment 24

[Cornel Ionce [:noni] [Hubs QA]]

Confirming that the number of times a user enters tablet mode is correctly updated in the Histograms (FX_TABLET_MODE_USED_DURING_SESSION).

Tested using Firefox 41 RC build 3 (build ID: 20150917150946) on a Surface Pro 2 device running Windows 10 64-bit.