1514006 - Implement telemetry for pocket + newtab

Status: VERIFIED FIXED

(Firefox :: New Tab Page, enhancement, P1)

Comment 1

[Nan Jiang [:nanj]]

We want to capture all the new metrics for the new Pocket + Newtab experience.

The proposed metrics are mostly based on the current telemetry for Top Stories, with an augment of following attributes:

* module ID (section ID)
* experiment information
  - experiment ID
  _ cohort ID
* user preferences

Comment 2

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/activity-stream/pull/4655

Comment 3

[[github robot]]

Commit pushed to master at https://github.com/mozilla/activity-stream

https://github.com/mozilla/activity-stream/commit/ac1cc87ef54471c554682e329382721eedd58d90
Bug 1514006 - Part 1, Add impression wrapper for Pocket sections (#4655)

Comment 4

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/activity-stream/pull/4666

Comment 5

[[github robot]]

Commit pushed to master at https://github.com/mozilla/activity-stream

https://github.com/mozilla/activity-stream/commit/e561898ae0fde451897a4b03c11627193d3003f9
Bug 1514006 - Part 2, add CLICK pings for the Hero and CardGrid (#4666)

Comment 6

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/activity-stream/pull/4671

Comment 7

[[github robot]]

Commit pushed to master at https://github.com/mozilla/activity-stream

https://github.com/mozilla/activity-stream/commit/62c87514b36d5fb46913224ebc64f7257a2a28cb
Bug 1514006 - Part 3, add impression and click pings for CardGrid and List (#4671)

Comment 8

[Ed Lee :Mardak]

https://hg.mozilla.org/mozilla-central/rev/25af8129d9f2

Comment 9

[Brahmini Nagabandi]

Hi,

Can you please provide QA steps.

Thanks.

Comment 10

[Ed Lee :Mardak]

https://hg.mozilla.org/mozilla-central/rev/381457de43c0

Comment 11

[Brahmini Nagabandi]

Attachment: QA Result : Bug - 1514006.png

QA Results :

Tested on :

FF Nightly version : 66.0a1 (2019-01-22)
OS : Mac and Windows 10 Pro

Note :
I did few rounds of testing with the default new tab version for the pings which has 1 row for Recommended by Pocket.

So, as part of testing I switched Recommended by Pocket rows from 1 to 2 and choose 5th card (on a 0-based index).

Below are the observations from Mac OS:

Impression ping :

Open New Tab, this has 6 cards in Recommended by Pocket sectio

TELEMETRY PING: {"locale":"en-US","topic":"activity-stream","client_id":"n/a","version":"66.0a1","release_channel":"nightly","addon_version":"20190122215349","user_prefs":31,"session_id":"n/a","page":"about:newtab","source":"TOP_STORIES","tiles":[{"id":32480},{"id":32414},{"id":32504},{"id":32469},{"id":32494},{"id":32516}],"action":"activity_stream_impression_stats","impression_id":"{0ac09703-cf38-be49-a470-c8c72565c022}","profile_creation_date":17833,"region":"US"}

JSON response :

{
"locale":"en-US",
"topic":"activity-stream",
"client_id":"n/a",
"version":"66.0a1",
"release_channel":"nightly",
"addon_version":"20190122215349",
"user_prefs":31,
"session_id":"n/a",
"page":"about:newtab",
"source":"TOP_STORIES",
"tiles":[
{
"id":32480
},
{
"id":32414
},
{
"id":32504
},
{
"id":32469
},
{
"id":32494
},
{
"id":32516
}
],
"action":"activity_stream_impression_stats",
"impression_id":"{0ac09703-cf38-be49-a470-c8c72565c022}",
"profile_creation_date":17833,
"region":"US"
}

Click ping :

Clicked on the 5th card (0-based index).

TELEMETRY PING: {"locale":"en-US","topic":"activity-stream","client_id":"n/a","version":"66.0a1","release_channel":"nightly","addon_version":"20190122215349","user_prefs":31,"session_id":"n/a","page":"about:newtab","source":"TOP_STORIES","click":0,"tiles":[{"id":32494,"pos":4}],"action":"activity_stream_impression_stats","impression_id":"{0ac09703-cf38-be49-a470-c8c72565c022}","profile_creation_date":17833,"region":"US"}

JSON Response :

{
"locale":"en-US",
"topic":"activity-stream",
"client_id":"n/a",
"version":"66.0a1",
"release_channel":"nightly",
"addon_version":"20190122215349",
"user_prefs":31,
"session_id":"n/a",
"page":"about:newtab",
"source":"TOP_STORIES",
"click":0,
"tiles":[
{
"id":32494,
"pos":4
}
],
"action":"activity_stream_impression_stats",
"impression_id":"{0ac09703-cf38-be49-a470-c8c72565c022}",
"profile_creation_date":17833,
"region":"US"
}

Quick question :

I do see "source":"TOP_STORIES" when I click from Recommended by Pocket - is this fine?
or do we expect "HERO", "CARDGRID", "LISTS" ?

Other than this everything is fine.

Comment 12

[Brahmini Nagabandi]

Synced with Najiang offline.

Per Najiang : All the existing Pocket telemetry metrics in release have source: TOP_STORIES, i believe this also applies to the control group of the MVP
for other MVP experiment variants, they should report source as one of HERO, CARDGRID, or LIST.

QA Note :
I also have the pref setting for browser.newtabpage.activity-stream.discoverystream.config = {"enabled":false,"layout_endpoint":"https://getpocket.com/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=control"}

So this is working as expected.

Closing the bug as verified.