1216227 - Tablet mode: count page loads inside/outside tablet mode to get a better idea of usage

Status: RESOLVED FIXED

(Firefox :: General, defect)

Description

[:Gijs (he/him)]

It seems like most people are in tablet mode twice per session ( https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2015-10-14&keys=__none__!__none__!__none__&max_channel_version=release%252F41&measure=FX_TABLET_MODE_USED_DURING_SESSION&min_channel_version=null&os=Windows_NT&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2015-09-17&table=0&trim=1&use_submission_date=0 )

Not only is that weird, it also doesn't really tell us if those sessions were accidents / super short, or if they were how they used the browser, etc.

The total number of non-zero submissions was roughly 1% compared with the total number of submissions for SIMPLEMEASURES_START on Windows 10, or about half as many submissions as we get for Linux (which is a little skewed because so many distros don't build with telemetry).

It'd be neat if we instead measured the number of pages loaded inside and outside telemetry, and that'd give us comparable measures.

Comment 1

[:Gijs (he/him)]

I'll look into this either tonight or tomorrow.

Comment 2

[:Gijs (he/him)]

Attachment: MozReview Request: Bug 1216227 - do bucketed page-load-per-window counts to assess tablet mode usage, r?MattN

Bug 1216227 - do bucketed page-load-per-window counts to assess tablet mode usage, r?MattN

Comment 3

[:Gijs (he/him)]

Comment on attachment 8676307 [details]
MozReview Request: Bug 1216227 - do bucketed page-load-per-window counts to assess tablet mode usage, r?MattN

Realized that this should probably be a keyed histogram instead...

Comment 4

[:Gijs (he/him)]

Comment on attachment 8676307 [details]
MozReview Request: Bug 1216227 - do bucketed page-load-per-window counts to assess tablet mode usage, r?MattN

Bug 1216227 - do bucketed page-load-per-window counts to assess tablet mode usage, r?MattN

Comment 5

[:Gijs (he/him)]

Comment on attachment 8676307 [details]
MozReview Request: Bug 1216227 - do bucketed page-load-per-window counts to assess tablet mode usage, r?MattN

Requesting privacy review from Vladan...

(In reply to Benjamin Smedberg  [:bsmedberg] from bug 731025 comment #34)
> Standard questions:
> * What are the questions that these probes are designed to address? Is this
> the minimum amount of data necessary to answer those questions?

How does active usage of tablet mode compare with active usage of desktop mode on windows 10 (with a view to: how much do we need to invest, comparatively, in integrating with tablet mode).

The probe counts location changes (page loads and tab switches) per window, and saves those in a keyed (desktop, tablet) exponentially bucketed telemetry histogram to be able to compare desktop and tablet mode usage.

> * Who is responsible for monitoring the data? How often will they do it? Do
> the necessary monitoring dashboards already exist or who is going to create
> them?

Dolske and me. I expect we'll check at least the release this lands on (hopefully 42, but if we can't make that 43) and the next few releases as we adjust various things (e.g. releasing proper on-screen keyboard support, hopefully with 43).

Note that this supersedes the existing probe which dolske and I looked at and realized wasn't telling us much, in particular, how long people are in/out of tablet mode and how they use it, meaning we couldn't really tell whether people actively used the browser at all in tablet mode, or not.

> * What's the user value?

If people frequently use Firefox in tablet mode, we will have evidence to warrant committing more resources to improving it, which in turn will benefit those users. If not, we put those resources somewhere else which will benefit all those people not using tablet mode... :-)

Comment 6

[Matthew N. [:MattN]]

Comment on attachment 8676307 [details]
MozReview Request: Bug 1216227 - do bucketed page-load-per-window counts to assess tablet mode usage, r?MattN

https://reviewboard.mozilla.org/r/22649/#review20149

::: browser/base/content/browser.js:5447
(Diff revision 2)
> +  inc() {

I find the "inc" name unclear. I think it means increment but I don't see a reason to abbreviate it here. It should be "increment" or "record" IMO.

::: browser/base/content/browser.js:5449
(Diff revision 2)
> +      this.inc = () => {};
> +      return;
> +    }
> +    this.inc = this._realInc;
> +    this.inc();

Hopefully this will make it fast enough that it won't affect Talos

::: browser/base/content/browser.js:5459
(Diff revision 2)
> +    let inTabletMode = document.documentElement.hasAttribute("tabletmode");
> +    this[inTabletMode ? "_tabletCount" : "_desktopCount"]++;

Are you checking the attribute instead of WindowsUIUtils.inTabletMode for performance reasons? I'm not familiar with this attribute and how it works so I'm assuming you tested this.

::: toolkit/components/telemetry/Histograms.json:4310
(Diff revision 2)
> -  "FX_TABLET_MODE_USED_DURING_SESSION": {
> -    "expires_in_version": "46",
> -    "kind": "count",
> -    "description": "Windows 10+ only: The number of times tablet-mode is used during a session"
> +  "FX_TABLETMODE_PAGE_LOAD": {
> +    "expires_in_version": "47",
> +    "kind": "exponential",
> +    "high": 100000,
> +    "n_buckets": 17,
> +    "keyed": true,

Repeating what I mentioned on IRC: If we only care about the proportion of tablet vs. desktop mode then we could use a "boolean" histogram instead. You would be able to see the distribution of page loads in the two buckets but I'm not sure how useful that will be.

::: toolkit/components/telemetry/Histograms.json:4310
(Diff revision 2)
> -  "FX_TABLET_MODE_USED_DURING_SESSION": {
> -    "expires_in_version": "46",
> -    "kind": "count",
> -    "description": "Windows 10+ only: The number of times tablet-mode is used during a session"
> +  "FX_TABLETMODE_PAGE_LOAD": {
> +    "expires_in_version": "47",
> +    "kind": "exponential",
> +    "high": 100000,
> +    "n_buckets": 17,
> +    "keyed": true,

Repeating what I mentioned on IRC: If we only care about the proportion of tablet vs. desktop mode then we could use a "boolean" histogram instead. You would be able to see the distribution of page loads in the two buckets but I'm not sure how useful that will be.

::: toolkit/components/telemetry/Histograms.json:4312
(Diff revision 2)
> -    "kind": "count",
> -    "description": "Windows 10+ only: The number of times tablet-mode is used during a session"
> +    "kind": "exponential",
> +    "high": 100000,

IMO this seems a bit high for top-level loads in a single session.

Comment 7

[Vladan Djeric (:vladan)]

https://reviewboard.mozilla.org/r/22649/#review20193

::: toolkit/components/telemetry/Histograms.json:4316
(Diff revision 2)
> +    "description": "Number of toplevel location changes in tablet and desktop mode (only used on win10 where tablet mode is available)"

I agree with MattN, your question is better answered with a boolean histogram.

To further simplify, you could just report whether the entire session was spent in tablet mode or desktop mode, possibly with a 3rd bucket for "mixed modes".

Do people really switch between tablet & desktop mode frequently?

Comment 8

[:Gijs (he/him)]

Comment on attachment 8676307 [details]
MozReview Request: Bug 1216227 - do bucketed page-load-per-window counts to assess tablet mode usage, r?MattN

(In reply to Vladan Djeric (:vladan) -- please needinfo! from comment #7)
> https://reviewboard.mozilla.org/r/22649/#review20193
> 
> ::: toolkit/components/telemetry/Histograms.json:4316
> (Diff revision 2)
> > +    "description": "Number of toplevel location changes in tablet and desktop mode (only used on win10 where tablet mode is available)"
> 
> I agree with MattN, your question is better answered with a boolean
> histogram.

Matt, dolske and I discussed this on IRC. The boolean histogram still won't tell us about comparative session length in either mode. Both dolske and I indicated that this information seemed useful, and as dolske noted at the time, if we went with the boolean histogram it is likely we would then want to switch to this type of histogram to get more details about our measurements. This bug is already one such step of "hm, those results are inconclusive, let's try to measure a bit more so that we can draw more sane conclusions". I'd rather not go through too many of those cycles.

> To further simplify, you could just report whether the entire session was
> spent in tablet mode or desktop mode, possibly with a 3rd bucket for "mixed
> modes".

We already measure something like this (this patch removes that measurement). The outcome was "about 1% of win10 users activate tablet mode at least once per session", but we don't know anything about usage in either mode. We have no idea how long the user is in tablet mode, and if they ended up there as an accident or not, and if Firefox was in the background or in active use while they were in tablet mode, or ... 

The telemetry this patch is adding solves those uncertainties by actually measuring a proxy for browser use for each mode.

> Do people really switch between tablet & desktop mode frequently?

Yes. You can do this by just attaching/detaching the keyboard on your surface pro, for instance. The existing telemetry indicates users of tablet mode tend to switch into tablet mode twice per session. Moreso than switch once (or start with it on). That's confusing, so we're trying to better understand that measurement. See comment #0.

Could you reconsider, please? :-)

Comment 9

[:Gijs (he/him)]

(In reply to Matthew N. [:MattN] from comment #6)
> Comment on attachment 8676307 [details]
> MozReview Request: Bug 1216227 - do bucketed page-load-per-window counts to
> assess tablet mode usage, r?MattN
> 
> https://reviewboard.mozilla.org/r/22649/#review20149
> 
> ::: browser/base/content/browser.js:5447
> (Diff revision 2)
> > +  inc() {
> 
> I find the "inc" name unclear. I think it means increment but I don't see a
> reason to abbreviate it here. It should be "increment" or "record" IMO.
> 
> ::: browser/base/content/browser.js:5449
> (Diff revision 2)
> > +      this.inc = () => {};
> > +      return;
> > +    }
> > +    this.inc = this._realInc;
> > +    this.inc();
> 
> Hopefully this will make it fast enough that it won't affect Talos

That's the plan, yes. :-)

> ::: browser/base/content/browser.js:5459
> (Diff revision 2)
> > +    let inTabletMode = document.documentElement.hasAttribute("tabletmode");
> > +    this[inTabletMode ? "_tabletCount" : "_desktopCount"]++;
> 
> Are you checking the attribute instead of WindowsUIUtils.inTabletMode for
> performance reasons?

yes.

> I'm not familiar with this attribute and how it works
> so I'm assuming you tested this.

I added it in bug 1164675.

> ::: toolkit/components/telemetry/Histograms.json:4312
> (Diff revision 2)
> > -    "kind": "count",
> > -    "description": "Windows 10+ only: The number of times tablet-mode is used during a session"
> > +    "kind": "exponential",
> > +    "high": 100000,
> 
> IMO this seems a bit high for top-level loads in a single session.

Tab switches also count for the tabbrowser's onLocationChange. We also know that some people have hundreds of tabs, and that on portable devices people often just sleep the device instead of rebooting it. That will lead to pretty long sessions. Those kinds of devices are pretty highly correlated with tablet mode to begin with. So I figured erring on the side of caution was appropriate.

Comment 10

[Vladan Djeric (:vladan)]

Comment on attachment 8676307 [details]
MozReview Request: Bug 1216227 - do bucketed page-load-per-window counts to assess tablet mode usage, r?MattN

https://reviewboard.mozilla.org/r/22649/#review20659

Reporting the # of tab switches and page-loads feels a little invasive, but it's bucketed pretty coarsely and we already report other per-page metrics

::: toolkit/components/telemetry/Histograms.json:4310
(Diff revision 2)
> -  "FX_TABLET_MODE_USED_DURING_SESSION": {
> -    "expires_in_version": "46",
> -    "kind": "count",
> -    "description": "Windows 10+ only: The number of times tablet-mode is used during a session"
> +  "FX_TABLETMODE_PAGE_LOAD": {
> +    "expires_in_version": "47",
> +    "kind": "exponential",
> +    "high": 100000,
> +    "n_buckets": 17,
> +    "keyed": true,

Just a reminder that this histogram will only be reported on Release-channel if the user opted into Telemetry

::: toolkit/components/telemetry/Histograms.json:4314
(Diff revision 2)
> +    "n_buckets": 17,

Just FYI, you're not going to get a lot of resolution on the lower end with 17 buckets and 100k max:

https://telemetry.mozilla.org/histogram-simulator/#low=1&high=100000&n_buckets=17&kind=linear&generate=normal

Comment 11

[:Gijs (he/him)]

(In reply to Vladan Djeric (:vladan) -- please needinfo! from comment #10)
> Comment on attachment 8676307 [details]
> MozReview Request: Bug 1216227 - do bucketed page-load-per-window counts to
> assess tablet mode usage, r?MattN
> 
> https://reviewboard.mozilla.org/r/22649/#review20659
> 
> Reporting the # of tab switches and page-loads feels a little invasive, but
> it's bucketed pretty coarsely and we already report other per-page metrics
> 
> ::: toolkit/components/telemetry/Histograms.json:4310
> (Diff revision 2)
> > -  "FX_TABLET_MODE_USED_DURING_SESSION": {
> > -    "expires_in_version": "46",
> > -    "kind": "count",
> > -    "description": "Windows 10+ only: The number of times tablet-mode is used during a session"
> > +  "FX_TABLETMODE_PAGE_LOAD": {
> > +    "expires_in_version": "47",
> > +    "kind": "exponential",
> > +    "high": 100000,
> > +    "n_buckets": 17,
> > +    "keyed": true,
> 
> Just a reminder that this histogram will only be reported on Release-channel
> if the user opted into Telemetry
> 
> ::: toolkit/components/telemetry/Histograms.json:4314
> (Diff revision 2)
> > +    "n_buckets": 17,
> 
> Just FYI, you're not going to get a lot of resolution on the lower end with
> 17 buckets and 100k max:
> 
> https://telemetry.mozilla.org/histogram-simulator/
> #low=1&high=100000&n_buckets=17&kind=linear&generate=normal

It's exponential, so I think this is the right link:

https://telemetry.mozilla.org/histogram-simulator/#low=1&high=100000&n_buckets=17&kind=exponential&generate=normal

Which looks OK to me on the low end? Maybe at the medium end it will be less useful... Can we update the number of buckets afterwards? If not, maybe we should use 30 from the start, to be conservative?

Comment 12

[Vladan Djeric (:vladan)]

You're right, I didn't set exponential in my simulator link. If the bucketing in your link is sufficient for you, then I don't have any objections.
You can't change bucketing parameters after deploying a histogram, you would have to create a new histogram. Feel free to use 30 buckets from the start if you need them.

Comment 13

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/83644216c190

Comment 14

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/83644216c190