1658787 - Save to Pocket button on save recs telemetry

Status: RESOLVED FIXED

(Firefox :: Pocket, enhancement, P1)

Description

[Scott [:thecount] Downe]

Right now we know we want to track:

1. impressions for seen on save recs.
2. click events for clicks on the on save recs.

Comment 1

[Kirill Demtchouk]

I spoke to Ian & Kenny. We landed on the following tracking approach to minimize data review:

1. Send events to track impressions, clicks, and saves

2. Send each event with the following parameters:
   -client_id
   -position
   -experiment (should be available in the response from RecIt, as per Ian's spec: https://docs.google.com/document/d/1TNavHbgW52yQfgSA3IZtBOZMujX8r0uzliS0SR2w5mA/edit#heading=h.6ywl0gym8cdi)

Comment 2

[Kirill Demtchouk]

Attachment: request.txt

Attaching Data Collection request for Kenny to review.

Comment 3

[klong]

Comment on attachment 9175015 [details]
request.txt

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Documentation of the new event is in this PR: https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/608

And will be available here when merged: https://github.com/mozilla-services/mozilla-pipeline-schemas/blob/master/schemas/activity-stream/on-save-recs/on-save-recs.1.schema.json

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. Users can opt-out the data collection by disabling the Firefox telemetry.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Kirill Demtchouk ([e-mail]) will be monitoring those metrics.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

This classified this as Category 2 because it tracks clicks/impressions of content recommendations, but it does not identify the content that was viewed/clicked.

Is the data collection request for default-on or default-off?

Default on for all channels and countries/locales.

Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

Does the data collection use a third-party collection tool?

No.

---

Result: r+

Comment 4

[Scott [:thecount] Downe]

Attachment: Bug 1658787 - On save rec telemetry.

Comment 5

[Scott [:thecount] Downe]

To test:

1. in about:config, set extensions.pocket.onSaveRecs to true.
2. Set browser.ping-centre.log to true
3. Restart
4. Open the multi process debugger (so you can see ping logs)
5. Open a newtab.
6. Open a newtab Pocket article.
7. Click the save to Pocket button for that article.
8. Log into a Pocket account.

Expected:

1. You should see a door hanger for the save to pocket button, including three link at the bottom.
2. You should see a log similar to this every time the door hanger opens:

TELEMETRY PING (pocket): {"experiments":{},"locale":"en-US","version":"82.0a1","release_channel":"default","model":"doc2vec-best-article-pubspread","events":[{"action":"impression","position":0},{"action":"impression","position":1},{"action":"impression","position":2}],"impression_id":"{nnnn}","addon_version":"20200914160713","profile_creation_date":18520,"region":"CA"}

3. You should see a log similar to this every time you click an article link, check the position to make sure it matches the link you clicked, 0-2:

TELEMETRY PING (pocket): {"experiments":{},"locale":"en-US","version":"82.0a1","release_channel":"default","model":"doc2vec-best-article-pubspread","events":[{"action":"click","position":2}],"impression_id":"{nnnn}","addon_version":"20200914160713","profile_creation_date":18520,"region":"CA"}

Comment 6

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/619

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4fe29fab5be4
On save rec telemetry. r=nanj

Comment 8

[Dorel Luca [:dluca]]

Backed out changeset 4fe29fab5be4 (bug 1658787) for Browser-chrome failures in browser/components/pocket/test/unit/browser_pocket_main.js. CLOSED TREE

Log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=315891848&repo=autoland&lineNumber=3556

Push that introduced failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=4fe29fab5be4eaf95f6f1b63ea200a4cb69b9070

Backout:
https://hg.mozilla.org/integration/autoland/rev/a0ab3744c3bec3983416b8af975329b033e29086

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/833cb5de7ec4
On save rec telemetry. r=nanj

Comment 10

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/833cb5de7ec4

Comment 11

[Scott [:thecount] Downe]

backout patch got fixed an re landed, I'm clearing the needinfo.