Closed Bug 1565709 Opened 6 months ago Closed 5 months ago

Add story engagement labels to story items

Categories

(Firefox :: New Tab Page, task, P1)

67 Branch
task
Points:
3

Tracking

()

RESOLVED FIXED
Firefox 70
Iteration:
70.3 - Aug 5 - 18
Tracking Status
firefox70 --- fixed

People

(Reporter: wolasi, Assigned: jcarlos)

References

(Blocks 2 open bugs)

Details

(Keywords: github-merged)

Attachments

(1 file)

Background
We want to label stories that have crossed an engagement threshold

  • Stories that have been recently and have no engagement data should be labelled as Fresh
  • Stories that achieve a certain level of engagement should be labelled as Popular

Goal
To see how these labels affect story engagement

Design Spec
https://www.figma.com/file/eGdZVYlZ5evvePHUCtKnYg7u/New-Tab-Spec-for-Fx-70?node-id=307%3A1

Flags: needinfo?(wkonu)

NI Mathijs, as view count needs to be supported in feeds. Thanks

Flags: needinfo?(mathijs)

You're right Punam. It's a non-trivial change server-side, but definitely something we could do. Let's scope and prioritize it during our planning meeting.

If we're going ahead we can branch bugs off of this one with server-side and client-side work.

Flags: needinfo?(mathijs)

I created a bug for the backend work https://bugzilla.mozilla.org/show_bug.cgi?id=1566707

Depends on: 1566707

Feed machine already sends this out, albeit on a debug field. It's probably being cleared out on the Web tier.

Example:

{
  "curated_rec_id": 40402,
  "debug": {
    "boosts": [
      {
        "in_score": 0.007218252641575964,
        "name": "age_penalty",
        "out_score": 0.007218252641575964,
        "penalty": 0.5
      },
      {
        "field": "domain_id",
        "field_value": 81870,
        "in_score": 0.007218252641575964,
        "min_spacing": 3,
        "name": "min_spacing",
        "out_score": 0.007218252641575964
      }
    ],
    "modelId": "b-0085-15k-60min",
    "modelType": "bayesian",
    "pseudoClicks": 119.34,
    "pseudoImprs": 14040,
    "pseudoLocale": "en-US",
    "realClicks": 69,
    "realImprs": 13735,
    "syndicated_resolved_id": null,
    "time_delta": 3600,
    "time_now": 1563554727,
    "time_since": 1563551127,
    "time_window_seconds": 3600,
    "total_clicks": 2768,
    "total_imprs": 1151096
  },
  "feed_item_id": "00713541-e3fc-45b1-b539-51aca3b981f6-r9eb48a5a",
  "pCTR": 0.007218252641575964,
  "post_id": 0,
  "rec_src": "CuratedLinksForLocaleSource",
  "resolved_id": 2662632026,
  "score": 0.007218252641575964
}
Assignee: nobody → mathijs
Iteration: --- → 70.2 - Jul 22 - Aug 4
Points: --- → 1
Priority: -- → P1
Assignee: mathijs → jcarlos
Points: 1 → 3
Flags: needinfo?(wkonu)
Summary: Add view count to story items → Add story engagement labels to story items

I've updated the description and added a spec

Blocked by backend work from Mathijs - Bug 1566707.

Mathijs do you have an ETA for this?

Flags: needinfo?(mathijs)

(In reply to jonathan koren from comment #4)

Feed machine already sends this out, albeit on a debug field. It's probably being cleared out on the Web tier.

Example:

{
  "curated_rec_id": 40402,
  "debug": {
    "boosts": [
      {
        "in_score": 0.007218252641575964,
        "name": "age_penalty",
        "out_score": 0.007218252641575964,
        "penalty": 0.5
      },
      {
        "field": "domain_id",
        "field_value": 81870,
        "in_score": 0.007218252641575964,
        "min_spacing": 3,
        "name": "min_spacing",
        "out_score": 0.007218252641575964
      }
    ],
    "modelId": "b-0085-15k-60min",
    "modelType": "bayesian",
    "pseudoClicks": 119.34,
    "pseudoImprs": 14040,
    "pseudoLocale": "en-US",
    "realClicks": 69,
    "realImprs": 13735,
    "syndicated_resolved_id": null,
    "time_delta": 3600,
    "time_now": 1563554727,
    "time_since": 1563551127,
    "time_window_seconds": 3600,
    "total_clicks": 2768,
    "total_imprs": 1151096
  },
  "feed_item_id": "00713541-e3fc-45b1-b539-51aca3b981f6-r9eb48a5a",
  "pCTR": 0.007218252641575964,
  "post_id": 0,
  "rec_src": "CuratedLinksForLocaleSource",
  "resolved_id": 2662632026,
  "score": 0.007218252641575964
}

Do you know what metric/threshold we could be using from here to determine "Fresh" or "Popular". Ideally we can house this logic on the server side.

Flags: needinfo?(wkonu)

(In reply to Jessilyn Davis from comment #6)

Blocked by backend work from Mathijs - Bug 1566707.

Mathijs do you have an ETA for this?

I expect we'll have all mission-critical AdZerk work done by Aug 9th, so if this is the next priority I could start around the Aug 12th, and it's just a day's worth of work.

Before I start we should make a decision on the above question of what metric(s) and threshold(s) we use.

Wolasi, have we already defined how we measure success? That'll be helpful in launching an experiment.

Flags: needinfo?(mathijs)

Re: metrics I thought we decided on number of clicks, and the popular threshold I was thinking the top 20-25th percentile for the Popular label. Fresh label would be stories that have no click data yet.

Re: measure of success, I'd consider not negatively impacting story engagement a success. But we should take this up with Kirill to see if we should run a formal A/B test, or some other tactic would suffice.

Flags: needinfo?(wkonu)

We would either want to run a formal AB test or (if we can assign different tile IDs to the same content with/without engagement labels) compare the click rate on the same content with/without labels.

The former would be preferred, but the latter would be faster (pending the tile ID point).

Iteration: 70.2 - Jul 22 - Aug 4 → 70.3 - Aug 5 - 18
See Also: → 1573905
Blocks: 1574334
Status: NEW → RESOLVED
Closed: 5 months ago
Keywords: github-merged
Resolution: --- → FIXED
Target Milestone: --- → Firefox 70
You need to log in before you can comment on or make changes to this bug.