1855811 - Implement telemetry event for sidebar displays without an ad available

Status: VERIFIED FIXED

(Firefox Graveyard :: Shopping, task, P2)

Description

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Implement the surfaceNoAdsAvailable probe as described in https://docs.google.com/spreadsheets/d/1o80xY2ssMs8sxEpfyo54YaOScWlonX62tcqDf90YkU8/edit#gid=0

Comment 1

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Attachment: Bug 1855811 - Implement telemetry event for sidebar displays without an ad available. r?perry.mcmanis

Depends on D190959

Comment 2

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Comment on attachment 9358364 [details]
Bug 1855811 - Implement telemetry event for sidebar displays without an ad available. r?perry.mcmanis

DATA REVIEW REQUEST

1. What questions will you answer with this data?

How often do we try to show an ad, but we don't show one, because there is no inventory available for a given product?

2. Why does Mozilla need to answer these questions? Are there benefits for users?
   Do we need this information to address product or business requirements?

This information will inform business decisions around ads.

3. What alternative methods did you consider to answer these questions?
   Why were they not sufficient?

There's no other way to gather this data.

4. Can current instrumentation answer these questions?

No.

5. List all proposed measurements and indicate the category of data collection for each
   measurement, using the Firefox data collection categories found on the Mozilla wiki.

Measurement Name	Measurement Description	Data Collection Category	Tracking Bug
shopping.surface_no_ads_available	On a supported product page, the review checker showed analysis, and review checker ads were enabled, but when we tried to fetch an ad from the ad server, no ad was available.	interaction	https://bugzilla.mozilla.org/show_bug.cgi?id=1855811

6. Please provide a link to the documentation for this data collection which
   describes the ultimate data set in a public, complete, and accurate way.

This collection is Glean so is documented in the Glean Dictionary.

7. How long will this data be collected?

This collection has expiry '122'.

8. What populations will you measure?

All channels, countries, and locales. No filters.

9. If this data collection is default on, what is the opt-out mechanism for users?

These collections are Glean. The opt-out can be found in the product's preferences.

10. Please provide a general description of how you will analyze this data.

Data Science will create dashboards.

11. Where do you intend to share the results of your analysis?

Internally.

12. Is there a third-party tool (i.e. not Glean or Telemetry) that you
    are proposing to use for this data collection?

No.

Comment 3

[Travis Long [:travis_]]

Comment on attachment 9358364 [details]
Bug 1855811 - Implement telemetry event for sidebar displays without an ad available. r?perry.mcmanis

Data Review

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, through the metrics.yaml file and the Glean Dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, through the data preferences in the application settings.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

N/A, collection to end or be renewed by version 122

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction data

5. Is the data collection request for default-on or default-off?

Default-on

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No

7. Is the data collection covered by the existing Firefox privacy notice?

Yes

8. Does the data collection use a third-party collection tool?

No

Result

data-review+

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/81fd3ecb5e45
Implement telemetry event for sidebar displays without an ad available. r=TravisLong

Comment 5

[Norisz Fay [:noriszfay]]

Backed out for causing bc failure on browser_shopping_ads_test.js

Backout link

Push with failures

Failure log

Comment 6

[:Gijs (he/him)]

Green try so I'm relanding this: https://treeherder.mozilla.org/jobs?repo=try&revision=8e6e95f7202a67a09d320190a97be2ab0712352b

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/15eef22279c4
Implement telemetry event for sidebar displays without an ad available. r=TravisLong

Comment 8

[Natalia Csoregi [:nataliaCs]]

Backed out for causing failures on browser_shopping_ad_not_available.js

• backout: https://hg.mozilla.org/integration/autoland/rev/636c436924a7e5772176b96580201dce74b40818
• push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&revision=15eef22279c46f4bc936bcf3b8292a65f2233d2d
• failure log: https://treeherder.mozilla.org/logviewer?job_id=433408868&repo=autoland&lineNumber=15554

[task 2023-10-23T12:42:57.531Z] 12:42:57     INFO - TEST-PASS | toolkit/components/shopping/test/browser/browser_shopping_ad_not_available.js | Sidebar should be visible. - true == true - 
[task 2023-10-23T12:42:57.531Z] 12:42:57     INFO - Waiting for sidebar to update.
[task 2023-10-23T12:42:57.531Z] 12:42:57     INFO - Product already loaded.
[task 2023-10-23T12:42:57.532Z] 12:42:57     INFO - Console message: [JavaScript Error: "Error: Can't find profile directory." {file: "resource://gre/modules/XULStore.sys.mjs" line: 60}]
[task 2023-10-23T12:42:57.532Z] 12:42:57     INFO - load@resource://gre/modules/XULStore.sys.mjs:60:15
[task 2023-10-23T12:42:57.532Z] 12:42:57     INFO - XULStore@resource://gre/modules/XULStore.sys.mjs:17:10
[task 2023-10-23T12:42:57.532Z] 12:42:57     INFO - 
[task 2023-10-23T12:42:57.532Z] 12:42:57     INFO - Console message: [JavaScript Error: "Error: Can't find profile directory." {file: "resource://gre/modules/XULStore.sys.mjs" line: 60}]
[task 2023-10-23T12:42:57.532Z] 12:42:57     INFO - load@resource://gre/modules/XULStore.sys.mjs:60:15
[task 2023-10-23T12:42:57.532Z] 12:42:57     INFO - XULStore@resource://gre/modules/XULStore.sys.mjs:17:10
[task 2023-10-23T12:42:57.532Z] 12:42:57     INFO - 
[task 2023-10-23T12:42:57.532Z] 12:42:57     INFO - Console message: [JavaScript Error: "Error: Can't find profile directory." {file: "resource://gre/modules/XULStore.sys.mjs" line: 60}]
[task 2023-10-23T12:42:57.533Z] 12:42:57     INFO - load@resource://gre/modules/XULStore.sys.mjs:60:15
[task 2023-10-23T12:42:57.533Z] 12:42:57     INFO - XULStore@resource://gre/modules/XULStore.sys.mjs:17:10
[task 2023-10-23T12:42:57.533Z] 12:42:57     INFO - 
[task 2023-10-23T12:42:57.533Z] 12:42:57     INFO - Buffered messages finished
[task 2023-10-23T12:42:57.537Z] 12:42:57     INFO - TEST-UNEXPECTED-FAIL | toolkit/components/shopping/test/browser/browser_shopping_ad_not_available.js | Should have recorded lack of ads. - "undefined" == 1 - {"filename":"chrome://mochitests/content/browser/toolkit/components/shopping/test/browser/browser_shopping_ad_not_available.js","name":"test_no_ad_available_telemetry/<","sourceId":605,"lineNumber":42,"columnNumber":12,"sourceLine":"","asyncCause":null,"asyncCaller":{"filename":"resource://testing-common/BrowserTestUtils.sys.mjs","name":"withNewTab","sourceId":588,"lineNumber":146,"columnNumber":22,"sourceLine":"","asyncCause":null,"asyncCaller":{"filename":"chrome://mochitests/content/browser/toolkit/components/shopping/test/browser/browser_shopping_ad_not_available.js","name":"test_no_ad_available_telemetry","sourceId":605,"lineNumber":24,"columnNumber":26,"sourceLine":"","asyncCause":null,"asyncCaller":null,"caller":{"filename":"chrome://mochikit/content/browser-test.js","name":"handleTask","sourceId":556,"lineNumber":1134,"columnNumber":26,"sourceLine":"","asyncCause":null,"asyncCaller":null,"caller":{"filename":"chrome://mochikit/content/browser-test.js","name":"_runTaskBasedTest","sourceId":556,"lineNumber":1206,"columnNumber":18,"sourceLine":"","asyncCause":null,"asyncCaller":{"filename":"chrome://mochikit/content/browser-test.js","name":"Tester_execTest","sourceId":556,"lineNumber":1348,"columnNumber":14,"sourceLine":"","asyncCause":null,"asyncCaller":null,"caller":{"filename":"chrome://mochikit/content/browser-test.js","name":"nextTest/<","sourceId":556,"lineNumber":1123,"columnNumber":14,"sourceLine":"","asyncCause":null,"asyncCaller":null,"caller":{"filename":"chrome://mochikit/content/tests/SimpleTest/SimpleTest.js","name":"SimpleTest.waitForFocus/<","sourceId":582,"lineNumber":1058,"columnNumber":13,"sourceLine":"","asyncCause":null,"asyncCaller":null,"caller":null,"formattedStack":"SimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13\n","nativeSavedFrame":{}},"formattedStack":"nextTest/<@chrome://mochikit/content/browser-test.js:1123:14\nSimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13\n","nativeSavedFrame":{}},"formattedStack":"async*Tester_execTest@chrome://mochikit/content/browser-test.js:1348:14\nnextTest/<@chrome://mochikit/content/browser-test.js:1123:14\nSimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13\n","nativeSavedFrame":{}},"caller":null,"formattedStack":"_runTaskBasedTest@chrome://mochikit/content/browser-test.js:1206:18\nasync*Tester_execTest@chrome://mochikit/content/browser-test.js:1348:14\nnextTest/<@chrome://mochikit/content/browser-test.js:1123:14\nSimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13\n","nativeSavedFrame":{}},"formattedStack":"handleTask@chrome://mochikit/content/browser-test.js:1134:26\n_runTaskBasedTest@chrome://mochikit/content/browser-test.js:1206:18\nasync*Tester_execTest@chrome://mochikit/content/browser-test.js:1348:14\nnextTest/<@chrome://mochikit/content/browser-test.js:1123:14\nSimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13\n","nativeSavedFrame":{}},"formattedStack":"async*test_no_ad_available_telemetry@chrome://mochitests/content/browser/toolkit/components/shopping/test/browser/browser_shopping_ad_not_available.js:24:26\nhandleTask@chrome://mochikit/content/browser-test.js:1134:26\n_runTaskBasedTest@chrome://mochikit/content/browser-test.js:1206:18\nasync*Tester_execTest@chrome://mochikit/content/browser-test.js:1348:14\nnextTest/<@chrome://mochikit/content/browser-test.js:1123:14\nSimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13\n","nativeSavedFrame":{}},"caller":null,"formattedStack":"async*withNewTab@resource://testing-common/BrowserTestUtils.sys.mjs:146:22\nasync*test_no_ad_available_telemetry@chrome://mochitests/content/browser/toolkit/components/shopping/test/browser/browser_shopping_ad_not_available.js:24:26\nhandleTask@chrome://mochikit/content/browser-test.js:1134:26\n_runTaskBasedTest@chrome://mochikit/content/browser-test.js:1206:18\nasync*Tester_execTest@chrome://mochikit/content/browser-test.js:1348:14\nnextTest/<@chrome://mochikit/content/browser-test.js:1123:14\nSimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13\n","nativeSavedFrame":{}},"caller":null,"formattedStack":"test_no_ad_available_telemetry/<@chrome://mochitests/content/browser/toolkit/components/shopping/test/browser/browser_shopping_ad_not_available.js:42:12\nasync*withNewTab@resource://testing-common/BrowserTestUtils.sys.mjs:146:22\nasync*test_no_ad_available_telemetry@chrome://mochitests/content/browser/toolkit/components/shopping/test/browser/browser_shopping_ad_not_available.js:24:26\nhandleTask@chrome://mochikit/content/browser-test.js:1134:26\n_runTaskBasedTest@chrome://mochikit/content/browser-test.js:1206:18\nasync*Tester_execTest@chrome://mochikit/content/browser-test.js:1348:14\nnextTest/<@chrome://mochikit/content/browser-test.js:1123:14\nSimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13\n","nativeSavedFrame":{}}
[task 2023-10-23T12:42:57.537Z] 12:42:57     INFO - Stack trace:
[task 2023-10-23T12:42:57.537Z] 12:42:57     INFO - chrome://mochitests/content/browser/toolkit/components/shopping/test/browser/browser_shopping_ad_not_available.js:test_no_ad_available_telemetry/<:42
[task 2023-10-23T12:42:57.537Z] 12:42:57     INFO - resource://testing-common/BrowserTestUtils.sys.mjs:withNewTab:146
[task 2023-10-23T12:42:57.537Z] 12:42:57     INFO - chrome://mochitests/content/browser/toolkit/components/shopping/test/browser/browser_shopping_ad_not_available.js:test_no_ad_available_telemetry:24
[task 2023-10-23T12:42:57.537Z] 12:42:57     INFO - chrome://mochikit/content/browser-test.js:handleTask:1134
[task 2023-10-23T12:42:57.537Z] 12:42:57     INFO - chrome://mochikit/content/browser-test.js:_runTaskBasedTest:1206
[task 2023-10-23T12:42:57.537Z] 12:42:57     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1348
[task 2023-10-23T12:42:57.537Z] 12:42:57     INFO - chrome://mochikit/content/browser-test.js:nextTest/<:1123
[task 2023-10-23T12:42:57.537Z] 12:42:57     INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058

Comment 9

[:Gijs (he/him)]

Having a whale of a time getting this test to pass... but fingers crossed the latest attempt does the trick.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/88862272a81e
Implement telemetry event for sidebar displays without an ad available. r=TravisLong

Comment 11

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/88862272a81e

Comment 12

[:Gijs (he/him)]

Attachment: Bug 1855811 - Implement telemetry event for sidebar displays without an ad available. r?perry.mcmanis

Original Revision: https://phabricator.services.mozilla.com/D190960

Comment 13

[Phabricator Automation]

Uplift Approval Request

• Steps to reproduce for manual QE testing: n/a
• Code covered by automated testing: yes
• User impact if declined: Missing data / telemetry information for something we're tracking for 120
• Is Android affected?: no
• String changes made/needed: no
• Risk associated with taking this patch: low
• Explanation of risk level: early in beta, 3 lines of JS code changed + metadata + automated test
• Needs manual QE test: no
• Fix verified in Nightly: no

Comment 14

[BugBot [:suhaib / :marco/ :calixte]]

Since nightly and release are affected, beta will likely be affected too.
For more information, please visit BugBot documentation.

Comment 15

[Dianna Smith [:diannaS]]

Comment on attachment 9359991 [details]
Bug 1855811 - Implement telemetry event for sidebar displays without an ad available. r?perry.mcmanis

Approved for 120.0b2

Comment 16

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/0e1ba587a1e6

Comment 17

[Rares Doghi, Desktop QA]

Verified as fixed in 120.0b2 as well as our latest Nightly build 121.0a1 (2023-10-25).