1566707 - Story engagement label Backend

Status: RESOLVED FIXED

(Firefox :: New Tab Page, task, P1)

Description

[Wolasi :wolasi]

Background
We want to label stories that have crossed an engagement threshold based on their click through rate.

• Stories that have been recently and have no engagement data should be labelled as Fresh
• Stories that achieve a certain level of engagement should be labelled as Popular

Let's determine what a good click rate threshold should be for the Popular label.

Goal
To see how these labels affect story engagement

Design Spec
https://www.figma.com/file/eGdZVYlZ5evvePHUCtKnYg7u/New-Tab-Spec-for-Fx-70?node-id=307%3A1

Comment 1

[Jessilyn Davis]

ETA to start on this Aug 12 (~1 day) - due Aug 13

Comment 2

[Punam Dahiya [:pdahiya]]

As described in front-end patch to display engagement, client expect a property engagement on an item. Its value should be of type String carrying exact text to be displayed in the card e.g "Fresh" or "Popular"

with recommendations JSON as
"recommendations": [
{
"id": 43816,
"url": "http://getpocket.com/...",
"domain": "theatlantic.com",
"engagement" : "Fresh",
"title": "Test",
....
},

https://github.com/mozilla/activity-stream/pull/5237#issue-305692887

Comment 3

[Mathijs Miermans]

Did we settle on a metric/threshold for displaying "Fresh" or "Popular".

I remember you saying you'd like the top 20-25% most-clicked items to be Popular and the rest Fresh?

I'll build out something and show it to you on Monday so you can QA.

Comment 4

[Wolasi :wolasi]

Popular: items in the top 20% for CTR

Let's skip the Fresh label for now.

Comment 5

[Jessilyn Davis]

From Mathijs on slack:
@jkoren is familiar with this codebase and could make this change if it's important enough to make it onto his plate. If not, my understanding is that the client-side changes are done and we can make the server-side change off-train when I'm back.

Jonathan, can you knock this out for us this month? Or do we need to wait until Oct when Mathijs is back?

Comment 6

[jonathan koren]

Yeah thats fine

Comment 7

[Jessilyn Davis]

Thanks, Jonathan!

Comment 8

[jonathan koren]

https://github.com/Pocket/Web/pull/2941

Comment 9

[jonathan koren]

Backend changes are on bug1566707.web.readitlater.com , and they'll hopefully go to prod Monday, Sept 16.

There's engagement field on each item. Its value is either empty string or popular. The returned items in the top 20% of CTR get the label.

Example popular record:

   {
      "id": 45532,
      "url": "https://getpocket.com/explore/item/meet-ladybeard-the-crown-prince-of-japan-s-strangest-music-scene?utm_source=pocket-newtab",
      "domain": "narratively.com",
      "title": "Meet Ladybeard, the Crown Prince of Japan’s Strangest Music Scene",
      "excerpt": "Richard Magarey has facial hair, pigtails, a closetful of schoolgirl outfits and an endless supply of energy. In Tokyo, he became the world’s u
nlikeliest death metal superstar.",
      "image_src": "https://img-getpocket.cdn.mozilla.net/direct?url=https%3A%2F%2Fpocket-image-cache.com%2F2000x%2Fhttps%253A%252F%252Fnarratively.com%252Fwp-co
ntent%252Fuploads%252F2018%252F10%252Fladybeard_HEADER_2580-522x310.jpg&resize=w450",
      "published_timestamp": "-62169984000",
      "engagement": "popular",

Example unpopular record:

    {
      "id": 45508,
      "url": "https://www.nytimes.com/interactive/2019/09/10/magazine/college-inequality.html?utm_source=pocket-newtab",
      "domain": "nytimes.com",
      "title": "I Was a Low-Income College Student. Classes Weren’t the Hard Part.",
      "excerpt": "Schools must learn that when you come from poverty, you need more than financial aid to succeed.",
      "image_src": "https://img-getpocket.cdn.mozilla.net/direct?url=https%3A%2F%2Fstatic01.nyt.com%2Fimages%2F2019%2F09%2F15%2Fmagazine%2F15mag-inequality%2F15m
ag-inequality-facebookJumbo.jpg&resize=w450",
      "published_timestamp": "1568088005",
      "engagement": "",

@jdavis Someone else will need to make the change in firefox to draw the actual label.

Comment 10

[Punam Dahiya [:pdahiya]]

UI for displaying these labels landed in nightly with PR https://github.com/mozilla/activity-stream/pull/5237. Fix displays engagement string value as is in UI.

Noticed in #comment 9 value of engagement field as all lower case 'popular'. If its easier we should send it from back-end with correct case e.g. 'Popular'. Thanks!

Comment 11

[jonathan koren]

Attachment: Screen Shot 2019-09-16 at 11.29.01 AM.png

Screenshot confirming deployment and labels

Comment 12

[jonathan koren]

Deployed. Labels show up in nightly. Screenshot attached.

Comment 13

[Jessilyn Davis]

Thanks @jkoren!

@marius/@ciprian, can y'all verify that this is showing up as it should in Beta 70? Design spec in #c0

Comment 14

[Jessilyn Davis]

Aaand per this slack thread [1] this isn't 100% ready to ship and we're late in the Beta 70 cycle.

@jkoren - can you back out this change?

We'll reassess with Mathijs is back vs. trying to tweak this to make it for 70.

[1] https://mozilla.slack.com/archives/CEZEF1AHX/p1568737325005800

Please let me know if you have any questions. Thanks!

Comment 15

[jonathan koren]

Sure https://github.com/Pocket/Web/pull/2949/

Comment 16

[jonathan koren]

jkoren-44283:~ jkoren$    curl -s 'https://getpocket.com/v3/firefox/global-recs/?useCache=false' -H 'authorization: Basic YWRtaW46bHV2NHBvY2tldA==' -H 'Origin: null' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.9' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36' -H 'content-type: application/json' -H 'Accept: */*' -H 'x-accept: application/json' -H 'Connection: keep-alive' --data-binary '{"guid":"40VTeA87gx60lr9b81d2a69B36p1g0ltsq4m18jR05K498TK742dtVbdwaKbDeP9","version":3,"locale_lang":"en-US","count":3,"access_token":null,"consumer_key":"70018-b83d4728573df682a7c50b3d", "feed_variant":"default_spocs_on", "count":27 }' |jq .|grep engagement

Is now returning nothing popular.