1522832 - consider fine grained visibility for impression reporting

Status: VERIFIED FIXED

(Firefox :: New Tab Page, enhancement, P1)

Description

[Nan Jiang [:nanj]]

The current Pocket impression ping is triggered on the page dom event visibilityState = "visble". This is sub-optimal for the new Pocket UI, since certain components could be invisible by default, hence an impression shouldn't be counted unless the user scrolls the page so that such item becomes visible.

We could use the intersection observer API to capture the individual visibility for each Pocket content, and make the impression reporting more accurate than its old implementation.

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/activity-stream/pull/4743

Comment 2

[Nan Jiang [:nanj]]

Hi Nick & Tushar, would like to hear your thoughts about the viewablity (visibility) of impression stats.

The current implementation for MVP re-uses the code of the existing Top Stories in release, which only checks the visibility status at the page level (i.e. Document.visitbilityState === "visible"). Apparently, this is not ideal for the new layouts where we can not fit everything in one page.

So we're looking to improve it by using the Intersection Observer API in order to make the impression reporting more accurate.

There are two ways to do this:

• We could do this at the card level, i.e. install an intersection observer for each story in every section. This will bring us the most accurate impression telemetry. However, the downside is that it has some performance impact because it will require lots of observers, and each observer needs to respond multiple visibility change events.

• The other approach is set up the observer at the component level, i.e. install an observer for each section (Hero, List, CardGrid etc.). With a lower overhead, it's also much simpler to implement. The downside is that it won't be as accurate as that of the card level one. It also requires us to define a visibility ratio for each section. It's common to use 50% as the threshold in the industry, but it won't work if that section crosses multiple page scrolls.

Still, I am inclined to pick the latter approach as a decent trade-off between accuracy and performance. We'd need to define some thresholds perhaps depending on the total number of cards in the section.

Let me know what you think.

Comment 3

[tushar]

Nan - does your proposed approach cover both organic and SPOC cards?

In the original New Tab data collection proposal (https://docs.google.com/document/d/1H7VAh2kc0Xhnl7GvWT64x5xTZNUc3r_hKYoM25N7XqA/) you'd sent over, I had made a comment about how capturing 'viewability' per the IAB would be preferred for SPOCs (and some links to Pocket / open source implementations:

1. Pocket's: https://github.com/Pocket/Web/blob/master/public_html/a/j/vendor/viewability.js
2. Open Source via an ad server company: https://github.com/picatcha/OpenAdViewability

So in addition to clarifying whether your approach is for both organic & SPOC cards, my question is whether we could implement the IAB standard only for SPOCs and maintain performance standards?

Lastly, given your comment about the limitations of implementing the observer at the component level for sections that cross multiple page scrolls, I wonder if we could have the viewability for organic content at a card level (but only for the first card in each component row) and maintain performance.

Anyhow, as it relates to SPOCs, my preference would be to implement at the card level, but for organic cards, I'd defer to Nick as to whether the limitation in your approach is OK.

Comment 4

[Nan Jiang [:nanj]]

(In reply to tushar from comment #3)

Nan - does your proposed approach cover both organic and SPOC cards?

Yes.

1. Pocket's: https://github.com/Pocket/Web/blob/master/public_html/a/j/vendor/viewability.js

I can't access this page.

2. Open Source via an ad server company: https://github.com/picatcha/OpenAdViewability

Yes, we use this as the reference.

So in addition to clarifying whether your approach is for both organic & SPOC cards, my question is whether we could implement the IAB standard only for SPOCs and maintain performance standards?

Yes, I think we can implement the IAB standard on SPOCS for sure. For organic cards, we apply a slightly lower standard (such as only check the 50% viewability) to minimize the performance impact.

Lastly, given your comment about the limitations of implementing the observer at the component level for sections that cross multiple page scrolls, I wonder if we could have the viewability for organic content at a card level (but only for the first card in each component row) and maintain performance.

Agreed, this also could be a backup option if the desired implementation introduces too much overhead.

I'd assume that we still prefer to have a card-level impression viewability other than component level. I think we can try with this first to see how it performs in practice.

Comment 5

[Nick Chapman]

Hey folks sorry I was confused about this bug since we have some of the visibility functionality working elsewhere. We do need this in 66 so that we're able to validate the click through rates for organic and sponsored content in our experiments on new tab. Without this we won't have any insights into the actual performance of content "below the fold".

Comment 6

[Nan Jiang [:nanj]]

[Tracking Requested - why for this release]:

Please see Comment 5 above.

Comment 7

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/activity-stream/pull/4798

Comment 8

[Ed Lee :Mardak]

QA steps:

1. about:config set…
   browser.newtabpage.activity-stream.telemetry = true
browser.ping-centre.log = true
   use dev-test-all (browser.newtabpage.activity-stream.discoverystream.config to {"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.com/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=dev-test-all"})
2. open Browser Console and search for activity_stream_impression_stats
3. open a new tab and slowly scroll down until you see "Left column Middle column Right column" then scroll so the middle column shows (1) but not (2)
4. close the tab
5. check Browser Console and there should be 1 entry in the log like…

TELEMETRY PING: … "source":"LIST","tiles":[{"id":33865},{"id":33865}],…

6. open new tab and scroll down this time to show in middle list (3) then close and check Browser Console for 2 entries:

TELEMETRY PING: … "source":"LIST","tiles":[{"id":33865},{"id":33865},{"id":33861},{"id":33861}…
TELEMETRY PING: … "source":"CARDGRID","tiles":[{"id":33865}],…

Might be good to copy/paste the entries from step 5 and 6 to report back

Comment 9

[Ed Lee :Mardak]

Attachment: Bug 1522832 - Fine-grained visibility on impression stats

Comment 10

[Ed Lee :Mardak]

https://hg.mozilla.org/mozilla-central/rev/1cfd69d05aa1

Comment 11

[Marius Coman [:mcoman], Ecosystem QA]

I have verified this issue with the latest Firefox Nightly (67.0a1 Build ID - 20190218094427) installed, on Windows 10 x64, Arch Linux and Mac 10.13.3. After I've followed the steps from comment 8 I've received the following telemetry pings:

1. The ping displayed at step 5 after I've closed the tab where only the first two stories were visible:

"TELEMETRY PING: "{"locale":"en-US","topic":"activity-stream","client_id":"n/a","version":"67.0a1","release_channel":"nightly","addon_version":"20190217214556","user_prefs":63,"session_id":"n/a","page":"about:newtab","source":"LIST","tiles":[{"id":33991},{"id":33991}],"action":"activity_stream_impression_stats","impression_id":"{b97bc38c-a1f8-4a3a-a99c-38028dd89ac9}","profile_creation_date":17945,"region":"US"}".

2. Pings displayed at step 6, after I've closed the tab where all the three stories were visible:

TELEMETRY PING: "{"locale":"en-US","topic":"activity-stream","client_id":"n/a","version":"67.0a1","release_channel":"nightly","addon_version":"20190217214556","user_prefs":63,"session_id":"n/a","page":"about:newtab","source":"LIST","tiles":[{"id":33991},{"id":33991},{"id":33978},{"id":33978},{"id":33950},{"id":33950}],"action":"activity_stream_impression_stats","impression_id":"{b97bc38c-a1f8-4a3a-a99c-38028dd89ac9}","profile_creation_date":17945,"region":"US"}"

TELEMETRY PING: "{"locale":"en-US","topic":"activity-stream","client_id":"n/a","version":"67.0a1","release_channel":"nightly","addon_version":"20190217214556","user_prefs":63,"session_id":"n/a","page":"about:newtab","source":"CARDGRID","tiles":[{"id":33991}],"action":"activity_stream_impression_stats","impression_id":"{b97bc38c-a1f8-4a3a-a99c-38028dd89ac9}","profile_creation_date":17945,"region":"US"}".

Comment 12

[Andrei Oprea [:andreio]]

Comment on attachment 9044377 [details]
Bug 1522832 - Fine-grained visibility on impression stats

Beta/Release Uplift Approval Request

Feature/Bug causing the regression

Bug 1512725

User impact if declined

Telemetry for the feature is affected

Is this code covered by automated tests?

Yes

Has the fix been verified in Nightly?

Yes

Needs manual test from QE?

No

If yes, steps to reproduce

Feature already tested

List of other uplifts needed

n/a

Risk to taking this patch

Medium

Why is the change risky/not risky? (and alternatives if risky)

Feature is in Nightly and has already been tested

String changes made/needed

Comment 13

[Scott [:thecount] Downe]

This also fixes bug 1523408 so we should test that in beta with these steps.

1. Set Discovery Stream pref "browser.newtabpage.activity-stream.discoverystream.config" to "{"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.com/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic"}"
2. Load a new tab.
3. Refresh that new tab.
4. Check and note browser.newtabpage.activity-stream.feeds.section.topstories.spoc.impressions
5. Refresh that tab again.
6. Check and note browser.newtabpage.activity-stream.feeds.section.topstories.spoc.impressions again.

Expected: That pref should either not exist, or should not be changing from steps 2, 3, and 5.
If it doesn't exist, it means you probably don't have any spoc impressions from previous non Discovery Stream usage, and if it does exist from previous usage, that's fine, but should not be changing with Discovery Stream on. It should instead be changing browser.newtabpage.activity-stream.discoverystream.spoc.impressions.

Comment 14

[Liz Henry (:lizzard) (relman/hg->git project)]

Double checked with jdavis, this should not need data review since it isn't collecting new data, just collecting it at a different point.

Comment 15

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 9044377 [details]
Bug 1522832 - Fine-grained visibility on impression stats

Fix for telemetry issue, OK to uplift as part of planned work on pocket/new tab for 66.
This should show up for beta 10.

Comment 16

[Dorel Luca [:dluca]]

https://hg.mozilla.org/releases/mozilla-beta/rev/64706783dc06

Comment 17

[Ed Lee :Mardak]

(In reply to Marius Coman [:mcoman], Experiments QA from comment #11)

I have verified this issue with the latest Firefox Nightly (67.0a1 Build ID - 20190218094427)

Could you please verify this issue on an early 66.0b10 build:

Windows 64: https://queue.taskcluster.net/v1/task/enVix5uvRveeQnNfxxl5mQ/runs/0/artifacts/public/build/target.zip
OS X: https://queue.taskcluster.net/v1/task/BbyPjtloRuCbHM2gX_9Hlw/runs/0/artifacts/public/build/target.dmg
Linux 64: https://queue.taskcluster.net/v1/task/CTU-aOfCSZe2B2tBVk_1GQ/runs/0/artifacts/public/build/target.tar.bz2

Treeherder: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&searchStr=64-n%2C%28n%29&revision=c5b7791033e051902646cf52f0bfd6946c5a1704

Comment 18

[Marius Coman [:mcoman], Ecosystem QA]

I have verified this issue with the latest Firefox Beta (66.0b10 Build ID - 20190219202808) installed, on Windows 10 x64, Arch Linux and Mac 10.13.3.

After I've followed the steps from comment 13 the "browser.newtabpage.activity-stream.feeds.section.topstories.spoc.impressions" was not created in the "about:config" page, however the "browser.newtabpage.activity-stream.discoverystream.spoc.impressions" pref's value was changed after refreshing the "about:newtab" page.