1565709 - Add story engagement labels to story items

Status: RESOLVED FIXED

(Firefox :: New Tab Page, task, P1)

Description

[Wolasi :wolasi]

Background
We want to label stories that have crossed an engagement threshold

• Stories that have been recently and have no engagement data should be labelled as Fresh
• Stories that achieve a certain level of engagement should be labelled as Popular

Goal
To see how these labels affect story engagement

Design Spec
https://www.figma.com/file/eGdZVYlZ5evvePHUCtKnYg7u/New-Tab-Spec-for-Fx-70?node-id=307%3A1

Comment 1

[Punam Dahiya [:pdahiya]]

NI Mathijs, as view count needs to be supported in feeds. Thanks

Comment 2

[Mathijs Miermans]

You're right Punam. It's a non-trivial change server-side, but definitely something we could do. Let's scope and prioritize it during our planning meeting.

If we're going ahead we can branch bugs off of this one with server-side and client-side work.

Comment 3

[Wolasi :wolasi]

I created a bug for the backend work https://bugzilla.mozilla.org/show_bug.cgi?id=1566707

Comment 4

[jonathan koren]

Feed machine already sends this out, albeit on a debug field. It's probably being cleared out on the Web tier.

Example:

{
  "curated_rec_id": 40402,
  "debug": {
    "boosts": [
      {
        "in_score": 0.007218252641575964,
        "name": "age_penalty",
        "out_score": 0.007218252641575964,
        "penalty": 0.5
      },
      {
        "field": "domain_id",
        "field_value": 81870,
        "in_score": 0.007218252641575964,
        "min_spacing": 3,
        "name": "min_spacing",
        "out_score": 0.007218252641575964
      }
    ],
    "modelId": "b-0085-15k-60min",
    "modelType": "bayesian",
    "pseudoClicks": 119.34,
    "pseudoImprs": 14040,
    "pseudoLocale": "en-US",
    "realClicks": 69,
    "realImprs": 13735,
    "syndicated_resolved_id": null,
    "time_delta": 3600,
    "time_now": 1563554727,
    "time_since": 1563551127,
    "time_window_seconds": 3600,
    "total_clicks": 2768,
    "total_imprs": 1151096
  },
  "feed_item_id": "00713541-e3fc-45b1-b539-51aca3b981f6-r9eb48a5a",
  "pCTR": 0.007218252641575964,
  "post_id": 0,
  "rec_src": "CuratedLinksForLocaleSource",
  "resolved_id": 2662632026,
  "score": 0.007218252641575964
}

Comment 5

[Wolasi :wolasi]

I've updated the description and added a spec

Comment 6

[Jessilyn Davis]

Blocked by backend work from Mathijs - Bug 1566707.

Mathijs do you have an ETA for this?

Comment 7

[Jeane Carlos]

(In reply to jonathan koren from comment #4)

Feed machine already sends this out, albeit on a debug field. It's probably being cleared out on the Web tier.

Example:

{
  "curated_rec_id": 40402,
  "debug": {
    "boosts": [
      {
        "in_score": 0.007218252641575964,
        "name": "age_penalty",
        "out_score": 0.007218252641575964,
        "penalty": 0.5
      },
      {
        "field": "domain_id",
        "field_value": 81870,
        "in_score": 0.007218252641575964,
        "min_spacing": 3,
        "name": "min_spacing",
        "out_score": 0.007218252641575964
      }
    ],
    "modelId": "b-0085-15k-60min",
    "modelType": "bayesian",
    "pseudoClicks": 119.34,
    "pseudoImprs": 14040,
    "pseudoLocale": "en-US",
    "realClicks": 69,
    "realImprs": 13735,
    "syndicated_resolved_id": null,
    "time_delta": 3600,
    "time_now": 1563554727,
    "time_since": 1563551127,
    "time_window_seconds": 3600,
    "total_clicks": 2768,
    "total_imprs": 1151096
  },
  "feed_item_id": "00713541-e3fc-45b1-b539-51aca3b981f6-r9eb48a5a",
  "pCTR": 0.007218252641575964,
  "post_id": 0,
  "rec_src": "CuratedLinksForLocaleSource",
  "resolved_id": 2662632026,
  "score": 0.007218252641575964
}

Do you know what metric/threshold we could be using from here to determine "Fresh" or "Popular". Ideally we can house this logic on the server side.

Comment 8

[Mathijs Miermans]

(In reply to Jessilyn Davis from comment #6)

Blocked by backend work from Mathijs - Bug 1566707.

Mathijs do you have an ETA for this?

I expect we'll have all mission-critical AdZerk work done by Aug 9th, so if this is the next priority I could start around the Aug 12th, and it's just a day's worth of work.

Before I start we should make a decision on the above question of what metric(s) and threshold(s) we use.

Wolasi, have we already defined how we measure success? That'll be helpful in launching an experiment.

Comment 9

[Wolasi :wolasi]

Re: metrics I thought we decided on number of clicks, and the popular threshold I was thinking the top 20-25th percentile for the Popular label. Fresh label would be stories that have no click data yet.

Re: measure of success, I'd consider not negatively impacting story engagement a success. But we should take this up with Kirill to see if we should run a formal A/B test, or some other tactic would suffice.

Comment 10

[Kirill Demtchouk]

We would either want to run a formal AB test or (if we can assign different tile IDs to the same content with/without engagement labels) compare the click rate on the same content with/without labels.

The former would be preferred, but the latter would be faster (pending the tile ID point).

Comment 11

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/activity-stream/pull/5237

Comment 12

[Ed Lee :Mardak]

https://hg.mozilla.org/mozilla-central/rev/cf52300e04c9