Closed Bug 1181500 Opened 5 years ago Closed 4 years ago

Instrumentation for tablet mode usage

Categories

(Firefox :: General, defect, P1)

Unspecified
Windows 10
defect

Tracking

()

VERIFIED FIXED
Firefox 43
Tracking Status
firefox41 - verified
firefox42 --- fixed
firefox43 --- fixed

People

(Reporter: phlsa, Assigned: jaws)

References

(Blocks 1 open bug)

Details

Attachments

(3 files, 2 obsolete files)

We have a bunch of tablet mode related issues in Firefox.
In order to have some sense of priority and severity of those bugs (as well as for prioritizing tablet mode specific new enhancements), we need some knowledge on how tablet mode is being used.

Here's the kinds of questions I think we need to get answered. Bryan, Dolske, feel free to edit.

- What % of sessions were fully or partially in tablet mode?
- How long are sessions in tablet mode (vs. normal session length)
- How often does the same user use tablet mode with Firefox? Another way of asking that is: do users continue using Firefox and tablet mode together after discovering it?

Since I assume that telemetry data would be too biased to be of any use here, this would probably need to be done in FHR.
Blocks: tabletmode
What's the process for getting this privacy-reviewed/approved/whatever, and are there examples of how to add this kind of data in FHR?

(also useful: how easy is implementing this on trunk and uplifting to 40, because FHR is going away but isn't dead in 40 yet?)
Flags: needinfo?(benjamin)
The generic process is documented at https://wiki.mozilla.org/Firefox/Data_Collection

Describe the data you want to collect in the form of a patch to the in-tree docs, and I or a peer can review them. Do you have docs that describe what tablet mode is? Is it a mode of Windows, or a mode of Firefox itself? Can a Firefox session switch between the two modes, and how common is that likely to be?

FHR is not a good match for the questions you're asking, because it doesn't typically collect session information (it aggregates by UTC day).

I don't think you're going to get this into Firefox 40 either way, so I recommend collecting this in unified telemetry, and perhaps adding it as part of the environment which will give us accurate correlations.
Flags: needinfo?(benjamin)
No longer blocks: windows-10
Blocks: 1191384
Attached patch Patch (obsolete) — Splinter Review
Looking at WindowsUIUtils.inTabletMode, it will return early if not on Windows 10 or later, so the inTabletMode value will always return `false` on non-Win10.

I didn't see a better group within TelemetryEnvironment so I put the data within the OS group. Mobile devices have `isTablet` in their DeviceData section.
Assignee: nobody → jaws
Status: NEW → ASSIGNED
Attachment #8650008 - Flags: review?(gijskruitbosch+bugs)
Attachment #8650008 - Attachment is obsolete: true
Attachment #8650008 - Flags: review?(gijskruitbosch+bugs)
Attachment #8650008 - Attachment is obsolete: false
Attachment #8650008 - Flags: review?(gijskruitbosch+bugs)
Comment on attachment 8650008 [details] [diff] [review]
Patch

For what it does, this code is fine.

However, I worry that this doesn't seem like it'll answer the questions in comment #0. This will just indicate whether we happened to be in or out of tablet mode at the time of the EnvironmentCache construction. We'll have no idea of whether users go into / out of tablet mode while Firefox is open, or if they continue using it like this (which is a hard question to answer anyway, because what constitutes "using" - does leaving it open in the background while doing something else count?)
Attachment #8650008 - Flags: review?(gijskruitbosch+bugs) → review+
Attached patch Patch for documentation update (obsolete) — Splinter Review
> Do you have docs that describe what tablet mode is? Is it a mode of Windows, or a mode of Firefox itself?

Tablet mode is a mode of Windows 10, which shows applications in fullscreen similar to Windows 8 Metro. Firefox may alter its behavior slightly when it finds that Tablet Mode is enabled.

> Can a Firefox session switch between the two modes, and how common is that likely to be?

Windows can switch between Tablet Mode enabled and disabled without restart. Some changes can be made without user interaction such as plugging in an external monitor (automatic disabling) or manually enabling/disabling via the Windows 10 Notification Area toggle.

We don't have data yet to know how common it is to switch between modes.
Attachment #8650564 - Flags: review?(benjamin)
Comment on attachment 8650564 [details] [diff] [review]
Patch for documentation update

I think it's perfectly find to record whether we're in tablet mode or not in telemetry.

It's not clear to me whether we should record this in the environment. If we do record this in the environment, we must keep it live up-to-date: a change to tablet-mode should cause a subsession split.

The advantage of causing a subsession split is that if you want to correlate other metrics, you can do so accurately. The major disadvantage is that subsession splits can be expensive and so we want to minimize them.

For now I'd recommend recording this as a boolean histogram named WINDOWS_IS_TABLETMODE: at startup, record a single value, and then if it changes record additional values. This won't get you correlations, but it should allow you to answer all of the questions from comment 0.
Attachment #8650564 - Flags: review?(benjamin) → review+
Comment on attachment 8650564 [details] [diff] [review]
Patch for documentation update

This is r+ because if you really do want to add this to the environment and do the splitting work, that's ok. But it seems like overkill and I'd recommend against it.
Attached patch Patch v2Splinter Review
This patch uses the "tablet-mode-change" observer notification and counts the number of times that tablet mode is entered (including if Firefox is started in tablet-mode) during a session.

We would like to monitor this data to understand how tablet mode is being used and to provide enough resources to support its usage. Users in tablet mode likely have different usage behaviors, such as potentially needing an on-screen keyboard and larger buttons, both of which have poor support for today.
Attachment #8650008 - Attachment is obsolete: true
Attachment #8650564 - Attachment is obsolete: true
Attachment #8656144 - Flags: review?(ttaubert)
Attachment #8656144 - Flags: feedback?(ally)
Comment on attachment 8656144 [details] [diff] [review]
Patch v2

Review of attachment 8656144 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM.
Attachment #8656144 - Flags: review?(ttaubert) → review+
Comment on attachment 8656144 [details] [diff] [review]
Patch v2

Review of attachment 8656144 [details] [diff] [review]:
-----------------------------------------------------------------

So the code as written counts the number of times we enter tablet mode. It does not include the session splitting that bsmedberg has mentioned, so you will not be able to draw correlations, including determining how long a user spent in tablet mode.  As long as you're cool with that, I'm cool with that.

To bring it back around to the original questions, I think you'll get enough data to start making some decisions.
Will this probe answer this question 
- What % of sessions were fully or partially in tablet mode? Yes
- How long are sessions in tablet mode (vs. normal session length) No
- How often does the same user use tablet mode with Firefox? Not solidly, but you'll get a pretty good idea. Telemetry is not very longitudinal.  

p=ally
Attachment #8656144 - Flags: feedback?(ally) → feedback+
(In reply to Allison Naaktgeboren please NEEDINFO? :ally from comment #10)
> Comment on attachment 8656144 [details] [diff] [review]
> Patch v2
> 
> Review of attachment 8656144 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> So the code as written counts the number of times we enter tablet mode. It
> does not include the session splitting that bsmedberg has mentioned, so you
> will not be able to draw correlations, including determining how long a user
> spent in tablet mode.  As long as you're cool with that, I'm cool with that.
> 
> To bring it back around to the original questions, I think you'll get enough
> data to start making some decisions.
> Will this probe answer this question 
> - What % of sessions were fully or partially in tablet mode? Yes
> - How long are sessions in tablet mode (vs. normal session length) No
> - How often does the same user use tablet mode with Firefox? Not solidly,
> but you'll get a pretty good idea. Telemetry is not very longitudinal.  
> 
> p=ally

Thanks, yes, this will be sufficient for now. We can iterate on this in the future if we feel we need more fine-grained data.
url:        https://hg.mozilla.org/integration/fx-team/rev/09ad75a2d51f2e8154c88da3b4d8a1951aa5f77a
changeset:  09ad75a2d51f2e8154c88da3b4d8a1951aa5f77a
user:       Jared Wein <jwein@mozilla.com>
date:       Thu Sep 03 14:34:29 2015 -0400
description:
Bug 1181500 - Instrumentation for tablet mode usage in desktop Firefox. r=ttaubert p=ally
[Tracking Requested - why for this release]: This is close. Adding the tracking flag for 41 per Ritu's request.
https://hg.mozilla.org/mozilla-central/rev/09ad75a2d51f
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 43
Approval Request Comment
[Feature/regressing bug #]: telemetry tracking for windows 10 features
[User impact if declined]: delayed data collection
[Describe test coverage new/current, TreeHerder]: manual verification of telemetry results
[Risks and why]: low risk, telemetry only
[String/UUID change made/needed]: none
Attachment #8657153 - Flags: approval-mozilla-aurora?
Attached patch Patch for Beta41Splinter Review
Approval Request Comment
[Feature/regressing bug #]: telemetry tracking for windows 10 features
[User impact if declined]: delayed data collection
[Describe test coverage new/current, TreeHerder]: manual verification of telemetry results
[Risks and why]: low risk, telemetry only
[String/UUID change made/needed]: none
Attachment #8657154 - Flags: approval-mozilla-beta?
Comment on attachment 8657153 [details] [diff] [review]
Patch for Aurora42

Additional telemetry data around tablet mode use is good and will help us to make the right data driven decisions. Aurora42+
Attachment #8657153 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Comment on attachment 8657154 [details] [diff] [review]
Patch for Beta41

Unified Telemetry is pref'd off by default in FF41. The risk associated with this uplift is therefore pretty low. Beta41+
Attachment #8657154 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
I wanted to track this in case the fix did not land soon enough so it would stay on my radar. Given that the fix has landed, I do not feel the need to track this. Please let me know if there are any concerns.
I accidentally landed the m-c version of the patch to aurora (didn't catch that there was an aurora-specific patch posted until after the fact). Backed out the original and landed the new one in ^
Flags: qe-verify+
Confirming that the number of times a user enters tablet mode is correctly updated in the Histograms (FX_TABLET_MODE_USED_DURING_SESSION).

Tested using Firefox 41 RC build 3 (build ID: 20150917150946) on a Surface Pro 2 device running Windows 10 64-bit.
Status: RESOLVED → VERIFIED
Flags: qe-verify+
QA Contact: cornel.ionce
Depends on: 1216227
You need to log in before you can comment on or make changes to this bug.