1546492 - QA test plan for discovery stream impressions tracking

Status: RESOLVED FIXED

(Firefox :: New Tab Page, task, P1)

Description

[Punam Dahiya [:pdahiya]]

Scope of this bug is to devise test plan for pocket new tab impressions tracking.

Comment 1

[Punam Dahiya [:pdahiya]]

Impression in discovery stream for recs and spoc items are tracked inside browser.newtabpage.activity-stream.discoverystream.rec.impressions and prefs browser.newtabpage.activity-stream.discoverystream.spoc.impressions respectively

Above prefs stores mapping of story(rec) IDs to a single timestamp (time of first impression). These timestamps guarantee a story doesn't stay on top for longer than configured in the feed settings (settings.recsExpireTime).

1. QA Steps to test discovery stream spocs impressions are not changing browser.newtabpage.activity-stream.feeds.section.topstories.spoc.impressions

Set discoverystream.config pref to {"api_key_pref":"extensions.pocket.oAuthConsumerKey","collapsible":true,"enabled":true,"show_spocs":true,"hardcoded_layout":false,"personalized":false,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic"}

https://bugzilla.mozilla.org/show_bug.cgi?id=1522832#c13

2. QA steps to test rotations of stories on the client

Set discoverystream.config pref to {"api_key_pref":"extensions.pocket.oAuthConsumerKey","collapsible":true,"enabled":true,"show_spocs":true,"hardcoded_layout":false,"personalized":false,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic"}

https://bugzilla.mozilla.org/show_bug.cgi?id=1527702#c3

3. QA steps to test spocs filtering by frequency cap

Set discoverystream.config pref to {"api_key_pref":"extensions.pocket.oAuthConsumerKey","collapsible":true,"enabled":true,"show_spocs":true,"hardcoded_layout":false,"personalized":false,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic"}

https://bugzilla.mozilla.org/show_bug.cgi?id=1520258#c6

Frequency cap for a spoc item is stored in feed configuration (e.g. https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=40249-e88c401e1b1f2242d9e441c4&locale_lang=en-US&feed_variant=default_spocs_on)

in format shown in example below which defines that for campaign 1 no more than 5 spocs can be shown in total lifetime, and no more than 2 per hour.

"campaign_id": 1,
"caps": {
"lifetime": 5,
"campaign": {
"count": 2,
"period": 3600
}
}

QA steps to test lifetime and period cap https://bugzilla.mozilla.org/show_bug.cgi?id=1546492#c3

QA steps to test spoc display by probability - https://bugzilla.mozilla.org/show_bug.cgi?id=1546492#c7

QA steps to test deduping spocs by domain - https://bugzilla.mozilla.org/show_bug.cgi?id=1546492#c8

QA steps for personalization:
For personalization v1 steps for spocs use this: https://github.com/mozilla/activity-stream/pull/4864#issuecomment-479962039

For personalization v1 steps for recs use this: https://github.com/mozilla/activity-stream/pull/4911#issuecomment-484588673

Comment 2

[Punam Dahiya [:pdahiya]]

NI Scott to review QA tests in above comment and if those suffice to give green to impression tracking working as expected for discovery stream.

We can add clearing cache resets impression and rec prefs for manual testing if needed. De-duping spocs by domain (https://bugzilla.mozilla.org/show_bug.cgi?id=1522899) manual testing will that be covered with personalization QA?

Comment 3

[Punam Dahiya [:pdahiya]]

QA steps to test lifetime and period frequency cap:

1. about:config and reset browser.newtabpage.activity-stream.discoverystream.spoc.impressions so it's {}, (from a clean profile, in browser.newtabpage.activity-stream.discoverystream.config layout_variant as basic and set hardcoded_layout as false)

2. open a new tab and note spoc story shown

3. check that the pref from step 1 is no longer just {} and is probably something like {"790":[1556134274653]} where 790 is campaign_id of the spoc story shown

4. Open JSON by using layout variant link from config
   e.g. https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=basic

Click on feed url link inside CardGrid component e.g. https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=40249-e88c401e1b1f2242d9e441c4&locale_lang=en-US&feed_variant=default_spocs_on

5. Open spoc item from spocs array with campaign id from step 3 and note values inside caps object

6. To test capping count within a period, refresh new tab to count given in caps->campaign -> count, time stamp should get appended to pref in step 1 for same campaign_id till the value seen in campaign count. After campaign count is met, spoc story should be replaced by a new spoc story.

7. To test the life time cap change clock to time more than campaign->period (in seconds)

8. restart firefox

9. initial home page probably is still showing the same stories (as it'll show cached stories while fetching the new ones)

10. refresh the page and spoc story in step 3 should show till lifetime count is met. Pref should be updated to reflect timestamp entries equivalent to lifetime count for that campaign_id. After lifetime count is met, spoc story should be replaced by a new spoc story.

Comment 4

[Scott [:thecount] Downe]

I'm going to run though those to make sure they still work, but they sound correct. So as long as I don't find any of these steps failing I think we're good.

Also personalization steps would not cover deduping spoc by domain, so yeah we should probably make a separate step of tests for that.

For personalization v1 steps for spocs use this: https://github.com/mozilla/activity-stream/pull/4864#issuecomment-479962039

For personalization v1 steps for recs use this: https://github.com/mozilla/activity-stream/pull/4911#issuecomment-484588673

I wonder if we also want steps for ensuring spoc rolls are the same even after the page renders because of context menu interaction.

Comment 5

[Scott [:thecount] Downe]

Oh and yeah, to answer your original question around spoc and rec impressions prefs, they are different.

Spocs impressions pref stores an array of the impressions as a timestamp, keyed on the spoc campaign id. This is used for frequency capping purposes. Example: {"1515":[1556549075312,1556549185433,1556549186560],"1529":[1556549186809,1556549187090]}

Recs impressions pref stores just the first impression as a timestanp keyed to rec id, and this is used for rotation. Example: {"35330":1556549075323,"36518":1556549186810,"37787":1556549075310,"37805":1556549075308}

The differences are:

1. spoc timestamps are an array of all impressions, and rec timestamps are just the first impression.
2. spocs are keyed on campaign id and recs are keyed on the main id. main ids are unique, campaign ids are not.
3. spocs are used for freq capping and recs for rotation.

Comment 6

[Scott [:thecount] Downe]

I think we're good though with the existing steps to test, they worked for me.

Comment 7

[Punam Dahiya [:pdahiya]]

Steps to verify Spoc stories display by probability (https://bugzilla.mozilla.org/show_bug.cgi?id=1535401#c3) and spoc rolls are the same even after the page renders because of context menu interaction.

1. Set discoverystream.config pref as

{"api_key_pref":"extensions.pocket.oAuthConsumerKey","collapsible":true,"enabled":true,"show_spocs":true,"hardcoded_layout":false,"personalized":false,"layout_endpoint":"https://gist.githubusercontent.com/punamdahiya/b4a0d9bb9dabac5e6248dd23ea5e09cf/raw/522fe988639b0ba890cf30bfcf6485f97566d592/feed-spocs.json"} and discoverystream.endpoints pref as 
https://getpocket.cdn.mozilla.net/,https://gist.githubusercontent.com/

2. Check pref browser.newtabpage.activity-stream.discoverystream.spoc.impressions and browser.newtabpage.activity-stream.discoverystream.rec.impressions are reset.

3. If needed reset pref browser.newtabpage.blocked and restart browser to make dismissed stories come back
.

4. Restart browser and ensure Card Grid component beside Hero component at the bottom of layout should have 3rd story (index 2) as sponsored content showing up with probability of 0.5. This is configured for CardGrid component (Component -> spocs -> probability) inside layout feed https://gist.githubusercontent.com/punamdahiya/b4a0d9bb9dabac5e6248dd23ea5e09cf/raw/522fe988639b0ba890cf30bfcf6485f97566d592/feed-spocs.json

Since rolled spocs pick random values, its possible for a spoc story to not display inside CardGrid component if rolled spoc is greater than 0.5. If that happens refresh tab to roll new spoc values.

5. In card grid component from step 4, click of dismiss link menu option in context menu on a non-spoc story should not cause disappear and reappear of 3rd story ensuring rolled spocs stays same between context menu dismiss clicks.

Comment 8

[Punam Dahiya [:pdahiya]]

Steps to test deduping spoc stories by domain

To test filtering out of campaign_duplicates we will be leveraging telemtry pings logged in browser console. Please verify telemetry logging in browserconsole is turned on by setting below prefs as true

• browser.ping-centre.log
• browser.newtabpage.activity-stream.telemetry
• browser.newtabpage.activity-stream.telemetry.structuredIngestion

1. Set discoverystream.config pref as

{"api_key_pref":"extensions.pocket.oAuthConsumerKey","collapsible":true,"enabled":true,"show_spocs":true,"hardcoded_layout":false,"personalized":false,"layout_endpoint":"https://gist.githubusercontent.com/punamdahiya/b4a0d9bb9dabac5e6248dd23ea5e09cf/raw/522fe988639b0ba890cf30bfcf6485f97566d592/feed-spocs.json"} and discoverystream.endpoints pref as
https://getpocket.cdn.mozilla.net/,https://gist.githubusercontent.com/

2. Check pref browser.newtabpage.activity-stream.discoverystream.spoc.impressions and browser.newtabpage.activity-stream.discoverystream.rec.impressions are reset

3. Restart browser and then open the browser console, look for the spoc_fills event in the output, you can add spoc_fills in the "filter output" section to filter out unrelated events

4. Verify in New Tab two sponsored stories are visible. Verify in browser console - telemetry ping similar to below showing two spoc stories displayed value as “1”

TELEMETRY PING (STRUCTURED INGESTION): {"locale":"en-US","client_id":"n/a","version":"68.0a1","release_channel":"default","addon_version":"20190423095327","user_prefs":255,"spoc_fills":
[{"id":38230,"reason":"n/a","displayed":1,"full_recalc":0},
{"id":27099,"reason":"n/a","displayed":1,"full_recalc":0}],

"impression_id":"{fffca6bc-224e-8643-869b-e267ea513976}","session_id":"n/a"}

5. Open spocs JSON feed, https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=40249-e88c401e1b1f2242d9e441c4&locale_lang=en-US&feed_variant=default_spocs_on to check campaign_id of displayed stories (38230 , 27099) from step 4. Searching is easier with Raw Data -> Pretty Print -> Cmd + F)

For id: 38230 camapaign_id should be 1666 and for id:27099 campaign_id is 1129

The same campaign_id should be recording impressions in browser.newtabpage.activity-stream.discoverystream.spoc.impressions pref

6. In opened spocs JSON feed from step 5, Search campaign_id '1666' and campaign_id '1129' for other stories id with similar domain

7. Open browser console and Verify id (other than 38230, 27099) located from step 6 are filtered out (displayed: 0) with reason as campaign_duplicates

{"reason":"campaign_duplicate","full_recalc":1,"id":38223,"displayed":0},{"reason":"campaign_duplicate","full_recalc":1,"id":38209,"displayed":0},{"reason":"campaign_duplicate","full_recalc":1,"id":38219,"displayed":0},{"reason":"campaign_duplicate","full_recalc":1,"id":38212,"displayed":0},{"reason":"campaign_duplicate","full_recalc":1,"id":38204,"displayed":0},{"reason":"campaign_duplicate","full_recalc":1,"id":29185,"displayed":0},{"reason":"campaign_duplicate","full_recalc":1,"id":29186,"displayed":0},{"reason":"campaign_duplicate","full_recalc":1,"id":29183,"displayed":0},{"reason":"campaign_duplicate","full_recalc":1,"id":29287,"displayed":0}],"impression_id":"{fffca6bc-224e-8643-869b-e267ea513976}","session_id":"n/a"}

Comment 9

[Punam Dahiya [:pdahiya]]

NI Scott to verify QA steps for spocs by probability and deduping spocs in https://bugzilla.mozilla.org/show_bug.cgi?id=1546492#c7 and https://bugzilla.mozilla.org/show_bug.cgi?id=1546492#c8 respectively. Thanks!

With these QA test cases, we have covered test plan for impression tracking and spocs display on pocket new tab

https://bugzilla.mozilla.org/show_bug.cgi?id=1546492#c1

Comment 10

[Scott [:thecount] Downe]

A few requested changes, otherwise looks good.

Comment 11

[Punam Dahiya [:pdahiya]]

Consolidated test plan as comment 1 for QA to use
https://bugzilla.mozilla.org/show_bug.cgi?id=1546492#c1

Updated text with changes and marking bug as resolved.