1377288 - Check AS telemetry for release blockers; make sure we can measure the things we want to measure

Status: RESOLVED WORKSFORME

(Firefox for Android Graveyard :: General, enhancement, P1)

Comment 1

[:Grisha Kruglov]

While the shape of data is similar to AS Desktop, absolute numbers do not really line up well. Three high level product metrics in particular stand out; here's my quick summary and some speculation:

1) Engagement rates on Android are ~30% lower than on Desktop.
2) Positive/negative interaction rate is about 40x on Desktop, and about 10x on Android. 
3) Highlights are much more likely to be interacted with on mobile vs. on desktop. Desktop Top Site interaction rate is ~25x that of highlights, while on Android that ratio is only about 3x.

(1) could be partially explained by different browsing patterns on mobile - users probably use "new tab" less frequently. It probably also indicates that mobile AS is in its current state inherently less useful than desktop's, and so we're seeing that it's being used less. I think doing a cross-check against classic "Top Sites" engagement rates would be a useful baseline.

Similarly, (2) might also indicate that users are probably seeing more noise in top sites/highlights on mobile. Alternatively, it could also indicate that users are more motivated to clean up their "new tab" experience on mobile, likely due to a very different form factor/context in which this UI resides.

(3) is particularly interesting, given that our "highlights" algorithms are very similar. Desktop does show more types of things (Android only has T.S. and highlights), but the vast difference is telling. ISTM that the most likely reason is that the "continuation" of use case is much more appealing on mobile. It would be very interesting to segment this metric by synced vs not-synced users, and we have the telemetry data for this in place already.

Another thing to note is that the dashboards have diverged, making direct comparisons harder. I'm wondering if there's value in committing to keeping up our telemetry dashboards for Desktop, Android and iOS in relative lockstep, particularly for the intersection of what we measure similarly across platforms. Android and iOS are inherently going to be closer neighbours due to how mobile AS Telemetry evolved.

NI Marina to double check that my numbers look correct and to comment on divergence of three different dashboards, and CC Maria for product visibility.

Comment 2

[:Grisha Kruglov]

Overall, it doesn't seem to me that anything in telemetry we're currently seeing underscore any new or existing blockers.

Comment 3

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

[triage] We're not going to add this to the current sprint but we should check in next sprint when we have more features.

Comment 4

[Chenxia Liu [:liuche]]

Hi Grisha, I know you're busy with bookmark sync so just checking in since we're 3 weeks out from A-S code freeze. Do you still have time to do this (some time in the next couple of weeks) or should we reassign to wrap things up?

As a note, Pocket Stories just merged this weekend so there won't be much telemetry from that right now, so this is something to do in the next few weeks. Maybe we also want to just keep this open throughout 58 too, to make sure there isn't a ton of unexpected fallout?

Comment 5

[:Grisha Kruglov]

I outlined my findings in Comment 1 from when I looked through this data. Comment 2 suggests that I didn't see anything outrageous. There are some interesting differences in how two A-S products (desktop/mobile) are used, but I haven't taken the time to translate that into something actionable.

My suggestion is for someone who's actively involved in A-S product to take over this. At this point I don't think this is an engineering bug, but rather a product exploration. Read through Comment 1 as a starting point, double check that the numbers are still current given all the changes that have been landing recently (I imagine we didn't change things fundamentally), and see if any of the findings might lead you to a useful product change/augmentation.

Comment 6

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

We need to make sure we have the probes to measure the things we want to measure.

Comment 7

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

My plan is as follows:

With telemetry, Maria wants to understand what the engagement [1] is *individually* for top sites, Pocket, and Highlights:
- How does it compare [2] with the old top sites?
- We should identify clicks as well as other interactions such as bookmarking and pinning

Note that we need to concretely define what [1] & [2] mean in terms of probes.

To gain context and execute, I plan to:
- See what data we collect for old top sites
- See what data we collect for AS on other platforms
- Define [1] & [2], try to be more concrete about the questions we want to answer, and verify my concrete questions with Maria
- Verify we have the probes, or land new ones, to collect the data we need
- (follow-up post merge?) Create dashboards to display this data

---

To quick address Grisha's work:

(In reply to :Grisha Kruglov from comment #1)
> 1) Engagement rates on Android are ~30% lower than on Desktop.
> 
> (1) could be partially explained by different browsing patterns on mobile -
> users probably use "new tab" less frequently. It probably also indicates
> that mobile AS is in its current state inherently less useful than
> desktop's, and so we're seeing that it's being used less. I think doing a
> cross-check against classic "Top Sites" engagement rates would be a useful
> baseline.

That's the plan! :) Let's hope we can figure out the discrepancies.

> Another thing to note is that the dashboards have diverged, making direct
> comparisons harder. I'm wondering if there's value in committing to keeping
> up our telemetry dashboards for Desktop, Android and iOS in relative
> lockstep, particularly for the intersection of what we measure similarly
> across platforms. Android and iOS are inherently going to be closer
> neighbours due to how mobile AS Telemetry evolved.

^ A combined dashboard sounds really useful – I definitely think we should consider this, provided we can ensure we're both collecting and reporting our metrics in comparable ways (I hope so, for even less strict comparisons sake!).

Note: there's more in comment 1 I did not address because I do not yet know how to.

Comment 8

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

> - See what data we collect for AS on other platforms

Here we go!

DESKTOP ---
dashboard [1]: I trust these are up to date. They have engagement (is this impressions? interactions?), retention (same questions), and specific interactions:
- top_stories
- save_to_pocket
- pin
- delete
- block
- top_sites
- highlights
- search

ANDROID ---
dashboard [2] [3] (yes, there are two!): I think we need to verify the accuracy of these dashboards but they look like they could be correct. Note that "proof-of-concept" is in one of the URLs. The first is similar to desktop: engagement, retention, and interactions (i.e. clicks on):
- highlights
- topsites
- pocket
- null (?)

The second dashboard is a comparison of positive vs. negative interactions - I don't know if this is still relevant because :emtwo only linked me the first dashboard for desktop so maybe they stopped using these numbers for desktop.

iOS ---
dashboard [4]: this looks like it needs updating, given that it doesn't split on pocket, highlights, and top sites.

---

Note: I'm still looking to see if we have an existing dashboard for old top sites.

[1]: https://sql.telemetry.mozilla.org/dashboard/activity-stream-system-addon-metrics-summary

[2]: https://sql.telemetry.mozilla.org/dashboard/android-activity-stream-proof-of-concept-
[3]: https://sql.telemetry.mozilla.org/dashboard/android-activity-stream-interactions

[4]: https://sql.telemetry.mozilla.org/dashboard/firefox-ios-metrics-summary

Comment 9

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

:emtwo also linked me to individual interaction graphs (top sites, pocket, etc.) for desktop with titles like "Average number of sessions per user" and "Average number of Top Story Clicks per User":

https://sql.telemetry.mozilla.org/dashboard/test-pilot-vs-nightly

Comment 10

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

(In reply to Michael Comella (:mcomella) from comment #7)
> - See what data we collect for old top sites

Barbara, are you aware of any existing dashboards for engagement-like queries on top sites, pre-Activity Stream?

From what I've found:
- Android: UI Telemetry Summary [1]: clicking the "Event summary" chart causes the browser to warn me a page is slowing down my computer, but I think this a similar query to...
- Android: UI Top Features [2]: this is a pivot table that shows the total number of clicks on various UI elements over the past 7 days
- Android: DAU (High Level by features) [3]. Shows unique daily users loading sites from bookmarks, top sites, etc.

To find these, I searched "android" and "fennec" and skimmed the list of the query search results.


[1]: https://sql.telemetry.mozilla.org/queries/59#633
[2]: https://sql.telemetry.mozilla.org/queries/2917#5504
[3]: https://sql.telemetry.mozilla.org/queries/169#272

Comment 11

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

To document some metrics we collect from the old top sites:
- Clicks on the list at the bottom [1]: load_url, list_item, top_sites
- Clicks on the grid at the top [2]: load_url, "suggestion" or "grid_item", "<position-#>" or "<position-#>-pinned"
- Pinning sites from the grid context menu [3]: pin
- Unpinning sites from the grid context menu [4]: unpin
- Editing sites from the grid context menu [5]: edit

The format is "Event, method, extra". This is for all telemetry in the TopSitesPanel file.

There is also telemetry that applies to all home panels (HomeFragment):
- A generic, "You clicked a context menu item" [6]: action, context_menu, <resource-id-for-action>, e.g. home_open_new_tab

With this generic telemetry item, we should be able to track *all* context menu clicks so there is no need for me to look at the individual telemetry items we might send in addition to the generic clicks (but we must be careful not to double-count the interactions when analyzing the telemetry!).

IMPORTANT NOTE: in old top sites, we may not be able to distinguish context menu clicks from the top sites panel vs. bookmarks vs. history. If there is a session for each panel, we could cross-reference against that, letting us distinguish these clicks, but that sounds challenging. I spoke with Maria and she mentioned that this lack of information is not important enough to uplift any patches for.

Note: I only looked in a few files and we we may collect more relevant telemetry that is attached from outside the panel (e.g. there is probably an event or session for displaying the TopSitesPanel, which would give us impressions).

[1]: https://dxr.mozilla.org/mozilla-beta/rev/6b5cce5da78594813192d46f129b6e5a012c9650/mobile/android/base/java/org/mozilla/gecko/home/TopSitesPanel.java#175
[2]: https://dxr.mozilla.org/mozilla-beta/rev/6b5cce5da78594813192d46f129b6e5a012c9650/mobile/android/base/java/org/mozilla/gecko/home/TopSitesPanel.java#227
[3]: https://dxr.mozilla.org/mozilla-beta/rev/6b5cce5da78594813192d46f129b6e5a012c9650/mobile/android/base/java/org/mozilla/gecko/home/TopSitesPanel.java#401
[4]: https://dxr.mozilla.org/mozilla-beta/rev/6b5cce5da78594813192d46f129b6e5a012c9650/mobile/android/base/java/org/mozilla/gecko/home/TopSitesPanel.java#416
[5]: https://dxr.mozilla.org/mozilla-beta/rev/6b5cce5da78594813192d46f129b6e5a012c9650/mobile/android/base/java/org/mozilla/gecko/home/TopSitesPanel.java#426

[6]: https://dxr.mozilla.org/mozilla-beta/rev/6b5cce5da78594813192d46f129b6e5a012c9650/mobile/android/base/java/org/mozilla/gecko/home/HomeFragment.java#274

Comment 12

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

> - Define [1] & [2], try to be more concrete about the questions we want to answer, and verify my concrete questions with Maria

I spoke with Maria. She emphasized that *accuracy* of the data is more important than the quantity of the data (i.e. how many probes we're collecting). Her priorities on the data we collect are:
- Clicks to load urls, divided by top sites, pocket, and highlights (already in the dashboard)
- Context menu interactions, divided by top sites, pocket, and highlights
- The telemetry on the Pocket More button, bug 1400408.

Like the other dashboards, it'd be good to include engagement and retention.

Notes:
- "engagement" = DAU/MAU, which is essentially an estimate on the percent of the MAU active per day.
- I have to verify the probes we already have in place are accurate

Questions:
1) Is collecting total number of interactions, and our engagement numbers, sufficient to determine the information we want to know? e.g. should we also be collecting impressions? fwiw, the desktop dashboard shows number of interactions per week.

2) The AS Android metrics for clicks to load urls includes "null" - why?

Comment 13

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

(In reply to Michael Comella (:mcomella) from comment #12)
> Questions:
> 1) Is collecting total number of interactions, and our engagement numbers,
> sufficient to determine the information we want to know? e.g. should we also
> be collecting impressions? fwiw, the desktop dashboard shows number of
> interactions per week.

Without following up with Maria, I think number we'd want to see, in order to understand how this value changes over time is: average total number of interactions, divided by type, per unique user, which we should be able to get from the data we collect.

Additionally, it's important that we're able to compare these interactions with old top sites and if we stick with these metrics, we should be able to because we're not adding additional data points.

Next step: verify, and land, the probes for the data we want to collect (comment 12). I filed bug 1401043 for actually creating the dashboards.

Comment 14

[:Grisha Kruglov]

A thought: you might want to account for ability to configure the A-S panel. Perhaps having an ability to segment analytics (and dashboards) by which portions of the UI are enabled would be helpful? I'm not sure if the telemetry part of that was done along with other AS Settings work.

Also, what does desktop do to account for such user customizations?

Comment 15

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Here's some of the telemetry we currently collect:
- Highlight or Pocket clicked [1]: load_url, list_item, {source_type = <highlights or pocket>, action_position = <position in list>, count = <total number of shown highlight or pocket items>}
- Top Site item clicked [2]: load_url, list_item, {source_type = top_sites, source_subtype = <pinned, suggested, top, blank (unused by AS)>, page_number = <page-number>, action_position = <position-in-grid>}
- Highlight or Pocket long-clicked or 3-dot settings clicked [3]: show, context_menu, <extras>
- Top sites long-clicked [4]: show, context_menu, <extras>

Note:
- The format is event, method, extras. The values I'm specifying are not the exact values (e.g. load_url is actually loadurl.1).

> 2) The AS Android metrics for clicks to load urls includes "null" - why?

The current code doesn't look like it can add "null" as the value for "source_type" - perhaps we execute the same telemetry action (load_url, list_item, ...) without a sourceType field in a different part of the code? Or the code at one point once allowed nulls but no longer does (e.g. before Pocket was added, what did we put there?)

Still TODO:
- Find and other telemetry we currently record (e.g. context menu items).
- Ensure we can distinguish which panel the context menu telemetry is launched from (since we no longer extend HomeFragment, this is probably easier because it's less generic)
- Verify all the extras for top sites, highlights, pocket are recorded correctly
- Check my results against the data on the server

[1]: http://searchfox.org/mozilla-central/rev/05c4c3bc0cfb9b0fc66bdfc8c47cac674e45f151/mobile/android/base/java/org/mozilla/gecko/activitystream/homepanel/StreamRecyclerAdapter.java#216
[2]: http://searchfox.org/mozilla-central/rev/1c13d5cf85f904afb8976c02a80daa252b893fca/mobile/android/base/java/org/mozilla/gecko/activitystream/homepanel/topsites/TopSitesPageAdapter.java#79

[3]: http://searchfox.org/mozilla-central/rev/05c4c3bc0cfb9b0fc66bdfc8c47cac674e45f151/mobile/android/base/java/org/mozilla/gecko/activitystream/homepanel/StreamRecyclerAdapter.java#299
[4]: http://searchfox.org/mozilla-central/rev/1c13d5cf85f904afb8976c02a80daa252b893fca/mobile/android/base/java/org/mozilla/gecko/activitystream/homepanel/topsites/TopSitesCard.java#81

Comment 16

[:Grisha Kruglov]

Activity Stream telemetry docs: http://searchfox.org/mozilla-central/source/mobile/android/docs/activitystreamtelemetry.rst

Please keep it up-to-date :-)

Comment 17

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Thanks for the docs, Grisha!

Instead of rewriting the docs as comments in this bug, I'll just step through the doc, verifying that everything in there still works and adding anything that's missing - my commit history will show what's incorrect or out-dated.

One thing that seems weird to me is how the `activitystream.1` session is started in setUserVisibleHint, rather than onResume, or similar, but in my testing, this seems to work correctly.

Comment 18

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

(In reply to :Grisha Kruglov from comment #14)
> A thought: you might want to account for ability to configure the A-S panel.

Good idea! I filed bug 1401404.

Comment 19

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

So the only problem I've found with the telemetry that could require changing the code (and is thus relevant to the merge date) is that clicking on a suggested top sites item the first time will say, "source_subtype = suggested" but subsequent presses will be "source_subtype = top". This isn't necessarily wrong but its unintuitive so we should either:
1) Document it
2) Change it so that a suggested site is always "subtype = suggested".

Comment 20

[:Grisha Kruglov]

Well... At what point does a visited suggested site change its state from "suggested" to a regular "top" site in a sparsely populated profile? Currently that happens whenever you visit it at least once.

I think just documenting this is caveat is fine.

I think this is only important when interpreting collected telemetry - that is, "suggested" clicks should be treated differently. 

With the current behavior, we can only ask "are suggested sites being used at all?" instead of "how frequently are suggested sites being used". Just make sure to document this someplace obvious for folks looking at the dashboards.

Comment 21

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

I completed my analysis that the probes all seem to be in place and working properly. In order to separate priorities, I filed bug 1402026 to update the docs to the current state of the code.

Comment 22

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Moved NI to bug 1401043.

Comment 23

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

(In reply to :Grisha Kruglov from comment #1)
> Another thing to note is that the dashboards have diverged, making direct
> comparisons harder. I'm wondering if there's value in committing to keeping
> up our telemetry dashboards for Desktop, Android and iOS in relative
> lockstep, particularly for the intersection of what we measure similarly
> across platforms. Android and iOS are inherently going to be closer
> neighbours due to how mobile AS Telemetry evolved.
> 
> NI Marina to double check that my numbers look correct and to comment on
> divergence of three different dashboards

I moved the NI for the divergence of the dashboards to bug 1401043 (build Android dashboards) but leaving the NI open for checking numbers (though I don't know how important that still is).

Comment 24

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Note: I could have been more thorough in this bug by verifying the data we see on the servers but:
1) Grisha has already done that to some extent
2) Our telemetry system has worked up until now and I verified that i) all the interactions logged their telemetry correctly in the Android console and ii) the code looks correct as well

And there are plenty more P1 issues to look at. :)