1402026 - Update Activity Stream telemetry documentation with missing details

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Activity Stream, enhancement, P2)

Description

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

In my telemetry analysis in bug 1377288, I noticed the documentation hasn't been kept up to date - this bug is to update the docs.

This isn't important to fix right now, but it'd be great to get in 58: P2 rank 1.

Comment 1

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Here is a link to the docs: http://searchfox.org/mozilla-central/source/mobile/android/docs/activitystreamtelemetry.rst

And here's one thing we'll have to document in this bug:

(In reply to Michael Comella (:mcomella) from comment #19)
> So the only problem I've found with the telemetry that could require
> changing the code (and is thus relevant to the merge date) is that clicking
> on a suggested top sites item the first time will say, "source_subtype =
> suggested" but subsequent presses will be "source_subtype = top". This isn't
> necessarily wrong but its unintuitive so we should either:
> 1) Document it
> 2) Change it so that a suggested site is always "subtype = suggested".

(In reply to :Grisha Kruglov from comment #20)
> Well... At what point does a visited suggested site change its state from
> "suggested" to a regular "top" site in a sparsely populated profile?
> Currently that happens whenever you visit it at least once.
> 
> I think just documenting this is caveat is fine.
> 
> I think this is only important when interpreting collected telemetry - that
> is, "suggested" clicks should be treated differently. 
> 
> With the current behavior, we can only ask "are suggested sites being used
> at all?" instead of "how frequently are suggested sites being used". Just
> make sure to document this someplace obvious for folks looking at the
> dashboards.

Comment 2

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Things to document:
- "suggested" -> "top" after one click from comment 1
- pocket and highlights have an extra, "interaction" = "menu_button" or "long_click" (or something like that)
- (via bug 1377288 comment 17) One thing that seems weird to me is how the `activitystream.1` session is started in setUserVisibleHint, rather than onResume, or similar, but in my testing, this seems to work correctly.

We could also restructure the docs because I think the current format is not ideal, but perhaps that's out of the scope of this bug. :)

Comment 3

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

[triage] agrees with analysis in comment 0.

Comment 4

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

All open Activity Stream bugs are moving from the whiteboard tag, "[mobileAS]", to the Firefox for Android component, "Activity Stream", so that I can keep better track of these bugs as the new triage owner; I will send out an email shortly with additional details, caveats, etc.

Comment 5

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

I should do this before I forget what exactly needs to be updated.

Comment 6

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Attachment: Bug 1402026: Add caveat about suggested source_subtype.

In the bug, we decided that it was okay to document this case because:
1) we didn't know the specific questions we were trying to answer
2) We were facing the 57 deadline

The alternative would be to change the behavior to perhaps the more intuitive
behavior where suggested sites will always be marked as "suggested" clicks but
note that there may be privacy concerns with that (in that there are a limited
number of suggested sites so we'd know the frequency that unique users might
visit the suggested sites).

Review commit: https://reviewboard.mozilla.org/r/184248/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/184248/

Comment 7

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Attachment: Bug 1402026: Correct top site context menu telemetry note.

The top sites menu button was removed in a previous iteration and long-click is
used to access the context menu now.

Review commit: https://reviewboard.mozilla.org/r/184250/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/184250/

Comment 8

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Attachment: Bug 1402026: Add interaction extra to AS telemetry.

Review commit: https://reviewboard.mozilla.org/r/184252/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/184252/

Comment 9

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Attachment: Bug 1402026: Correct formatting for AS telemetry bulleted lists.

Apparently, if you don't have whitespace above the bulleted list, it won't
format as a list.

Review commit: https://reviewboard.mozilla.org/r/184254/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/184254/

Comment 10

[Chenxia Liu [:liuche]]

Comment on attachment 8912932 [details]
Bug 1402026: Add caveat about suggested source_subtype.

https://reviewboard.mozilla.org/r/184248/#review189424

Documentation clarification, seems fine.

Comment 11

[Chenxia Liu [:liuche]]

Comment on attachment 8912933 [details]
Bug 1402026: Correct top site context menu telemetry note.

https://reviewboard.mozilla.org/r/184250/#review189426

Comment 12

[Chenxia Liu [:liuche]]

Comment on attachment 8912934 [details]
Bug 1402026: Add interaction extra to AS telemetry.

https://reviewboard.mozilla.org/r/184252/#review189428

::: mobile/android/docs/activitystreamtelemetry.rst:157
(Diff revision 1)
>      extras: {
>          ...
>          "count": number /* total number of highlights displayed */
>      }
>  
> +For "show.1" event, the following extra information is also recorded:

Can you also update the "real example" at the end of this document to be consistent? I believe there's only one "show" event.

Comment 13

[Chenxia Liu [:liuche]]

Comment on attachment 8912935 [details]
Bug 1402026: Correct formatting for AS telemetry bulleted lists.

https://reviewboard.mozilla.org/r/184254/#review189430

Comment 14

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Comment on attachment 8912934 [details]
Bug 1402026: Add interaction extra to AS telemetry.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/184252/diff/1-2/

Comment 15

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Comment on attachment 8912935 [details]
Bug 1402026: Correct formatting for AS telemetry bulleted lists.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/184254/diff/1-2/

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9e6eb09000d3
Add caveat about suggested source_subtype. r=liuche
https://hg.mozilla.org/integration/autoland/rev/55432b5c3cd8
Correct top site context menu telemetry note. r=liuche
https://hg.mozilla.org/integration/autoland/rev/cd12463f9043
Add interaction extra to AS telemetry. r=liuche
https://hg.mozilla.org/integration/autoland/rev/aece6300bbea
Correct formatting for AS telemetry bulleted lists. r=liuche

Comment 17

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/9e6eb09000d3
https://hg.mozilla.org/mozilla-central/rev/55432b5c3cd8
https://hg.mozilla.org/mozilla-central/rev/cd12463f9043
https://hg.mozilla.org/mozilla-central/rev/aece6300bbea