consider fine grained visibility for impression reporting

VERIFIED FIXED in Firefox 66

Status

()

enhancement
P1
normal
VERIFIED FIXED
7 months ago
2 months ago

People

(Reporter: nanj, Assigned: nanj)

Tracking

({github-merged})

unspecified
Firefox 67
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox66+ verified, firefox67 verified)

Details

Attachments

(3 attachments)

The current Pocket impression ping is triggered on the page dom event visibilityState = "visble". This is sub-optimal for the new Pocket UI, since certain components could be invisible by default, hence an impression shouldn't be counted unless the user scrolls the page so that such item becomes visible.

We could use the intersection observer API to capture the individual visibility for each Pocket content, and make the impression reporting more accurate than its old implementation.

Blocks: 1512725
Assignee: nobody → najiang
Severity: normal → enhancement
Iteration: --- → 67.1 - Jan 28 - Feb 10
Priority: -- → P1

Hi Nick & Tushar, would like to hear your thoughts about the viewablity (visibility) of impression stats.

The current implementation for MVP re-uses the code of the existing Top Stories in release, which only checks the visibility status at the page level (i.e. Document.visitbilityState === "visible"). Apparently, this is not ideal for the new layouts where we can not fit everything in one page.

So we're looking to improve it by using the Intersection Observer API in order to make the impression reporting more accurate.

There are two ways to do this:

  • We could do this at the card level, i.e. install an intersection observer for each story in every section. This will bring us the most accurate impression telemetry. However, the downside is that it has some performance impact because it will require lots of observers, and each observer needs to respond multiple visibility change events.

  • The other approach is set up the observer at the component level, i.e. install an observer for each section (Hero, List, CardGrid etc.). With a lower overhead, it's also much simpler to implement. The downside is that it won't be as accurate as that of the card level one. It also requires us to define a visibility ratio for each section. It's common to use 50% as the threshold in the industry, but it won't work if that section crosses multiple page scrolls.

Still, I am inclined to pick the latter approach as a decent trade-off between accuracy and performance. We'd need to define some thresholds perhaps depending on the total number of cards in the section.

Let me know what you think.

Flags: needinfo?(tushar)
Flags: needinfo?(nchapman)

Nan - does your proposed approach cover both organic and SPOC cards?

In the original New Tab data collection proposal (https://docs.google.com/document/d/1H7VAh2kc0Xhnl7GvWT64x5xTZNUc3r_hKYoM25N7XqA/) you'd sent over, I had made a comment about how capturing 'viewability' per the IAB would be preferred for SPOCs (and some links to Pocket / open source implementations:

  1. Pocket's: https://github.com/Pocket/Web/blob/master/public_html/a/j/vendor/viewability.js
  2. Open Source via an ad server company: https://github.com/picatcha/OpenAdViewability

So in addition to clarifying whether your approach is for both organic & SPOC cards, my question is whether we could implement the IAB standard only for SPOCs and maintain performance standards?

Lastly, given your comment about the limitations of implementing the observer at the component level for sections that cross multiple page scrolls, I wonder if we could have the viewability for organic content at a card level (but only for the first card in each component row) and maintain performance.

Anyhow, as it relates to SPOCs, my preference would be to implement at the card level, but for organic cards, I'd defer to Nick as to whether the limitation in your approach is OK.

Flags: needinfo?(tushar)

(In reply to tushar from comment #3)

Nan - does your proposed approach cover both organic and SPOC cards?

Yes.

  1. Pocket's: https://github.com/Pocket/Web/blob/master/public_html/a/j/vendor/viewability.js

I can't access this page.

  1. Open Source via an ad server company: https://github.com/picatcha/OpenAdViewability

Yes, we use this as the reference.

So in addition to clarifying whether your approach is for both organic & SPOC cards, my question is whether we could implement the IAB standard only for SPOCs and maintain performance standards?

Yes, I think we can implement the IAB standard on SPOCS for sure. For organic cards, we apply a slightly lower standard (such as only check the 50% viewability) to minimize the performance impact.

Lastly, given your comment about the limitations of implementing the observer at the component level for sections that cross multiple page scrolls, I wonder if we could have the viewability for organic content at a card level (but only for the first card in each component row) and maintain performance.

Agreed, this also could be a backup option if the desired implementation introduces too much overhead.

I'd assume that we still prefer to have a card-level impression viewability other than component level. I think we can try with this first to see how it performs in practice.

Iteration: 67.1 - Jan 28 - Feb 10 → 67.2 - Feb 11 - 24

Hey folks sorry I was confused about this bug since we have some of the visibility functionality working elsewhere. We do need this in 66 so that we're able to validate the click through rates for organic and sponsored content in our experiments on new tab. Without this we won't have any insights into the actual performance of content "below the fold".

Flags: needinfo?(nchapman)

[Tracking Requested - why for this release]:

Please see Comment 5 above.

Blocks: 1528410

QA steps:

  1. about:config set…
    browser.newtabpage.activity-stream.telemetry = true
    browser.ping-centre.log = true
    use dev-test-all (browser.newtabpage.activity-stream.discoverystream.config to {"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.com/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=dev-test-all"})
  2. open Browser Console and search for activity_stream_impression_stats
  3. open a new tab and slowly scroll down until you see "Left column Middle column Right column" then scroll so the middle column shows (1) but not (2)
  4. close the tab
  5. check Browser Console and there should be 1 entry in the log like…
TELEMETRY PING: … "source":"LIST","tiles":[{"id":33865},{"id":33865}],…
  1. open new tab and scroll down this time to show in middle list (3) then close and check Browser Console for 2 entries:
TELEMETRY PING: … "source":"LIST","tiles":[{"id":33865},{"id":33865},{"id":33861},{"id":33861}…
TELEMETRY PING: … "source":"CARDGRID","tiles":[{"id":33865}],…

Might be good to copy/paste the entries from step 5 and 6 to report back

Status: NEW → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 67

I have verified this issue with the latest Firefox Nightly (67.0a1 Build ID - 20190218094427) installed, on Windows 10 x64, Arch Linux and Mac 10.13.3. After I've followed the steps from comment 8 I've received the following telemetry pings:

  1. The ping displayed at step 5 after I've closed the tab where only the first two stories were visible:

"TELEMETRY PING: "{"locale":"en-US","topic":"activity-stream","client_id":"n/a","version":"67.0a1","release_channel":"nightly","addon_version":"20190217214556","user_prefs":63,"session_id":"n/a","page":"about:newtab","source":"LIST","tiles":[{"id":33991},{"id":33991}],"action":"activity_stream_impression_stats","impression_id":"{b97bc38c-a1f8-4a3a-a99c-38028dd89ac9}","profile_creation_date":17945,"region":"US"}".

  1. Pings displayed at step 6, after I've closed the tab where all the three stories were visible:

TELEMETRY PING: "{"locale":"en-US","topic":"activity-stream","client_id":"n/a","version":"67.0a1","release_channel":"nightly","addon_version":"20190217214556","user_prefs":63,"session_id":"n/a","page":"about:newtab","source":"LIST","tiles":[{"id":33991},{"id":33991},{"id":33978},{"id":33978},{"id":33950},{"id":33950}],"action":"activity_stream_impression_stats","impression_id":"{b97bc38c-a1f8-4a3a-a99c-38028dd89ac9}","profile_creation_date":17945,"region":"US"}"

TELEMETRY PING: "{"locale":"en-US","topic":"activity-stream","client_id":"n/a","version":"67.0a1","release_channel":"nightly","addon_version":"20190217214556","user_prefs":63,"session_id":"n/a","page":"about:newtab","source":"CARDGRID","tiles":[{"id":33991}],"action":"activity_stream_impression_stats","impression_id":"{b97bc38c-a1f8-4a3a-a99c-38028dd89ac9}","profile_creation_date":17945,"region":"US"}".

Status: RESOLVED → VERIFIED

Comment on attachment 9044377 [details]
Bug 1522832 - Fine-grained visibility on impression stats

Beta/Release Uplift Approval Request

Feature/Bug causing the regression

Bug 1512725

User impact if declined

Telemetry for the feature is affected

Is this code covered by automated tests?

Yes

Has the fix been verified in Nightly?

Yes

Needs manual test from QE?

No

If yes, steps to reproduce

Feature already tested

List of other uplifts needed

n/a

Risk to taking this patch

Medium

Why is the change risky/not risky? (and alternatives if risky)

Feature is in Nightly and has already been tested

String changes made/needed

Attachment #9044377 - Flags: approval-mozilla-beta?

This also fixes bug 1523408 so we should test that in beta with these steps.

  1. Set Discovery Stream pref "browser.newtabpage.activity-stream.discoverystream.config" to "{"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.com/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic"}"
  2. Load a new tab.
  3. Refresh that new tab.
  4. Check and note browser.newtabpage.activity-stream.feeds.section.topstories.spoc.impressions
  5. Refresh that tab again.
  6. Check and note browser.newtabpage.activity-stream.feeds.section.topstories.spoc.impressions again.

Expected: That pref should either not exist, or should not be changing from steps 2, 3, and 5.
If it doesn't exist, it means you probably don't have any spoc impressions from previous non Discovery Stream usage, and if it does exist from previous usage, that's fine, but should not be changing with Discovery Stream on. It should instead be changing browser.newtabpage.activity-stream.discoverystream.spoc.impressions.

Double checked with jdavis, this should not need data review since it isn't collecting new data, just collecting it at a different point.

Comment on attachment 9044377 [details]
Bug 1522832 - Fine-grained visibility on impression stats

Fix for telemetry issue, OK to uplift as part of planned work on pocket/new tab for 66.
This should show up for beta 10.

Attachment #9044377 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

I have verified this issue with the latest Firefox Beta (66.0b10 Build ID - 20190219202808) installed, on Windows 10 x64, Arch Linux and Mac 10.13.3.

After I've followed the steps from comment 13 the "browser.newtabpage.activity-stream.feeds.section.topstories.spoc.impressions" was not created in the "about:config" page, however the "browser.newtabpage.activity-stream.discoverystream.spoc.impressions" pref's value was changed after refreshing the "about:newtab" page.

Flags: needinfo?(marius.coman)
Component: Activity Streams: Newtab → New Tab Page
You need to log in before you can comment on or make changes to this bug.