1245368 - Add telemetry for home panel customizations

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Awesomescreen, defect)

Description

[Chenxia Liu [:liuche]]

We should add a Simple Measures probe to see how many users have customized what home panels are displayed.

Comment 1

[Chenxia Liu [:liuche]]

Attachment: MozReview Request: Bug 1245368 - Add telemetry for home panel customizations. r=margaret

Review commit: https://reviewboard.mozilla.org/r/33351/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/33351/

Comment 2

[Chenxia Liu [:liuche]]

Comment on attachment 8715118 [details]
MozReview Request: Bug 1245368 - Add telemetry for home panel customizations. r=margaret

I looked at how Desktop adds simple measures - I'm not entirely sure this will work because it's also calling into Java because the config is stored in SharedPreferences, and it has to wait for a callback.

Is there some way to trigger a telemetry collection?

Comment 3

[Mark Finkle (:mfinkle) (use needinfo?)]

(In reply to Chenxia Liu [:liuche] from comment #2)
> Comment on attachment 8715118 [details]
> MozReview Request: Bug 1245368 - Add telemetry for home panel customizations.
> 
> I looked at how Desktop adds simple measures - I'm not entirely sure this
> will work because it's also calling into Java because the config is stored
> in SharedPreferences, and it has to wait for a callback.

Do you have a way to visualize the Simple Measures? Is there a dashboard somewhere for that?

> Is there some way to trigger a telemetry collection?

Putting Fennec into the background triggers a telemetry collection

Comment 4

[Mark Finkle (:mfinkle) (use needinfo?)]

We also have some UI Telemetry for hiding, showing, moving, and setting default. Look in Events for "panel.*.1" events. Looks like most activity on hiding panels or setting different defaults.

Comment 5

[:Margaret Leibovic]

Why are we using UI telemtry? Can we just use a boolean histogram to know whether or not users have a stored panel config?

If I remember our conversation correctly, we talked about just wanting to understand what % of users would require a home config migration. To do that, I would just create a new histogram probe and do a check here to see if that customized home config shared preference exists: 
http://hg.mozilla.org/mozilla-central/file/tip/mobile/android/base/java/org/mozilla/gecko/BrowserApp.java#l1727

Comment 6

[:Margaret Leibovic]

Comment on attachment 8715118 [details]
MozReview Request: Bug 1245368 - Add telemetry for home panel customizations. r=margaret

Like mfinkle said, we don't have a way to visualize this, so I don't think we should do this. It also seems like too much overhead to add a new JSON message for the sake of gathering this data.

Comment 7

[Chenxia Liu [:liuche]]

(In reply to :Margaret Leibovic from comment #5)
> Why are we using UI telemtry? Can we just use a boolean histogram to know
> whether or not users have a stored panel config?
> 
Hm, I remember us talking about *not* using histograms since histograms are are suited to collecting a range of values (like startup times) rather than a single value (is pref X enabled). And with a histogram, we won't have the "end state," just how many times they've sent telemetry pings while in each state (which could still be helpful), whereas simple measures reflects a snapshot of the most recent state.

And I'm not sure about this but if the histograms are aggregated on the histograms dashboard, I don't think we'd be able to tell the _number_ of users who have each enabled, but would get the total number of responses we've collected, with multiple responses from each user. I suppose we could use this as a proxy for "number of people who have customized home panels", or switch our question to "what proportion of the time do people have customized home panels".

Comment 8

[:Margaret Leibovic]

(In reply to Chenxia Liu [:liuche] from comment #7)
> (In reply to :Margaret Leibovic from comment #5)
> > Why are we using UI telemtry? Can we just use a boolean histogram to know
> > whether or not users have a stored panel config?
> > 
> Hm, I remember us talking about *not* using histograms since histograms are
> are suited to collecting a range of values (like startup times) rather than
> a single value (is pref X enabled). And with a histogram, we won't have the
> "end state," just how many times they've sent telemetry pings while in each
> state (which could still be helpful), whereas simple measures reflects a
> snapshot of the most recent state.

We already use histograms for the "is pref X enabled" state, if you look at the code I point to above.

> And I'm not sure about this but if the histograms are aggregated on the
> histograms dashboard, I don't think we'd be able to tell the _number_ of
> users who have each enabled, but would get the total number of responses
> we've collected, with multiple responses from each user. I suppose we could
> use this as a proxy for "number of people who have customized home panels",
> or switch our question to "what proportion of the time do people have
> customized home panels".

I'm fine with this proxy level, since it's something that we already do for the other prefs we measure.

Bottom line is that we need a way to visualize the telemetry that we include, and I think it would be too much work to make a new simple measurements dashboard just for this use case.

Comment 9

[Chenxia Liu [:liuche]]

Comment on attachment 8715118 [details]
MozReview Request: Bug 1245368 - Add telemetry for home panel customizations. r=margaret

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/33351/diff/1-2/

Comment 10

[:Margaret Leibovic]

Comment on attachment 8715118 [details]
MozReview Request: Bug 1245368 - Add telemetry for home panel customizations. r=margaret

https://reviewboard.mozilla.org/r/33351/#review30445

::: mobile/android/base/java/org/mozilla/gecko/BrowserApp.java:1752
(Diff revision 2)
> +            Telemetry.addToHistogram("FENNEC_HOMEPANELS_CUSTOM", hasCustomHomepanels ? 1 : 0);

This seems fine to me, but you'll need to define a new histogram in Histograms.json.

Comment 11

[Chenxia Liu [:liuche]]

Comment on attachment 8715118 [details]
MozReview Request: Bug 1245368 - Add telemetry for home panel customizations. r=margaret

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/33351/diff/2-3/

Comment 12

[:Margaret Leibovic]

Comment on attachment 8715118 [details]
MozReview Request: Bug 1245368 - Add telemetry for home panel customizations. r=margaret

https://reviewboard.mozilla.org/r/33351/#review30675

Comment 13

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/e342ec93ddcb

Comment 14

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/e342ec93ddcb