1889404 - Add basic telemetry for Content Relevancy

Status: RESOLVED FIXED

(Application Services :: Relevancy, task)

Comment 1

[Nan Jiang [:nanj]]

Attachment: Bug 1889404 - Add basic telemetry for content relevancy r=bdk,ttran

Comment 2

[Nan Jiang [:nanj]]

Attachment: data_review_request_bug_1889404.txt

Hi :chutten, could you take a look at this, please?

For context, we have already secured the legal sign-off for the validation phase (see "Legal Sign-offs" in the RFC).

Note that for the validation phase, we would only verify various things for client-side user interest classification without using the results to rank any content in the browser.

Also cc: Elizabeth for your awareness.

Comment 3

[Nan Jiang [:nanj]]

Attachment: data_review_request_bug_1889404.txt

Hi Sergio, thanks for the review!

I've made some minor tweaks per code review comments (the last three measurements under Q5), could you please take another look at it and re-grant the data-review+? Thanks!

Comment 4

[Sergio [:sergio]]

Comment on attachment 9398163 [details]
data_review_request_bug_1889404.txt

DATA COLLECTION REVIEW RESPONSE:
​
    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?
​
Yes, Glean.
​
    Is there a control mechanism that allows the user to turn the data collection on and off?
​
Yes. This collection can be controlled through the product's preferences.
​
    If the request is for permanent data collection, is there someone who will monitor the data over time?
​
Yes, [e-mail] and [e-mail] will be responsible.
​
    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
​
Category 2, Interaction.
​
    Is the data collection request for default-on or default-off?
​
Default on for all channels and countries.
​
    Does the instrumentation include the addition of any new identifiers?
​
No.
​
    Is the data collection covered by the existing Firefox privacy notice?
​
Yes.
​
    Does the data collection use a third-party collection tool?
​
No.
​
---
Result: datareview+

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ac231730f6c6
Add basic telemetry for content relevancy r=bdk

Comment 6

[Sandor Molnar[:smolnar]]

Backed out for causing bc failures @ browser_contentrelevancy_telemetry

Backout link: https://hg.mozilla.org/integration/autoland/rev/66a537a0f752a8b187e501f571a2c661d304ae9e

Push with failures

Failure log -> TEST-UNEXPECTED-FAIL | toolkit/components/contentrelevancy/tests/browser/browser_contentrelevancy_telemetry.js | Uncaught exception in test bound test_classify_succeed

TEST-PASS | toolkit/components/contentrelevancy/tests/browser/browser_contentrelevancy_telemetry.js | "undefined" == null - 
[task 2024-04-25T21:54:38.653Z] 21:54:38     INFO - Buffered messages finished
[task 2024-04-25T21:54:38.655Z] 21:54:38     INFO - TEST-UNEXPECTED-FAIL | toolkit/components/contentrelevancy/tests/browser/browser_contentrelevancy_telemetry.js | Uncaught exception in test bound test_classify_succeed - at chrome://mochitests/content/browser/toolkit/components/contentrelevancy/tests/browser/browser_contentrelevancy_telemetry.js:52 - TypeError: can't access property 0, Glean.relevancyClassify.succeed.testGetValue() is null
[task 2024-04-25T21:54:38.655Z] 21:54:38     INFO - Stack trace:
[task 2024-04-25T21:54:38.655Z] 21:54:38     INFO - test_classify_succeed@chrome://mochitests/content/browser/toolkit/components/contentrelevancy/tests/browser/browser_contentrelevancy_telemetry.js:52:37
[task 2024-04-25T21:54:38.655Z] 21:54:38     INFO - async*handleTask@chrome://mochikit/content/browser-test.js:1139:26
[task 2024-04-25T21:54:38.655Z] 21:54:38     INFO - _runTaskBasedTest@chrome://mochikit/content/browser-test.js:1211:18
[task 2024-04-25T21:54:38.655Z] 21:54:38     INFO - Tester_execTest@chrome://mochikit/content/browser-test.js:1353:14
[task 2024-04-25T21:54:38.655Z] 21:54:38     INFO - nextTest/<@chrome://mochikit/content/browser-test.js:1128:14
[task 2024-04-25T21:54:38.655Z] 21:54:38     INFO - SimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13
[task 2024-04-25T21:54:38.656Z] 21:54:38     INFO - Leaving test bound test_classify_succeed
[task 2024-04-25T21:54:38.656Z] 21:54:38     INFO - Entering test bound test_classify_fail_case1
[task 2024-04-25T21:55:24.791Z] 21:55:24     INFO - Not taking screenshot here: see the one that was previously logged
[task 2024-04-25T21:55:24.793Z] 21:55:24     INFO - TEST-UNEXPECTED-FAIL | toolkit/components/contentrelevancy/tests/browser/browser_contentrelevancy_telemetry.js | Test timed out - 
[task 2024-04-25T21:55:24.794Z] 21:55:24     INFO - GECKO(7440) | Completed ShutdownLeaks collections in process 7020
[task 2024-04-25T21:55:24.795Z] 21:55:24     INFO - TEST-START | Shutdown

Comment 7

[Nan Jiang [:nanj]]

Retry landing with a potential fix.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1bfb1774ed03
Add basic telemetry for content relevancy r=bdk

Comment 9

[Sandor Molnar[:smolnar]]

Backed out for causing bc failures @ browser_contentrelevancy_telemetry

Backout link: https://hg.mozilla.org/integration/autoland/rev/7139810235147a4159299e10f84a3cba9f53a0af

Push with failures

Failure log -> TEST-UNEXPECTED-FAIL | toolkit/components/contentrelevancy/tests/browser/browser_contentrelevancy_telemetry.js | Uncaught exception in test bound test_classify_succeed

 Buffered messages finished
[task 2024-04-26T02:54:33.465Z] 02:54:33     INFO - TEST-UNEXPECTED-FAIL | toolkit/components/contentrelevancy/tests/browser/browser_contentrelevancy_telemetry.js | Uncaught exception in test bound test_classify_succeed - at chrome://mochitests/content/browser/toolkit/components/contentrelevancy/tests/browser/browser_contentrelevancy_telemetry.js:57 - TypeError: can't access property 0, Glean.relevancyClassify.succeed.testGetValue() is null
[task 2024-04-26T02:54:33.465Z] 02:54:33     INFO - Stack trace:
[task 2024-04-26T02:54:33.465Z] 02:54:33     INFO - test_classify_succeed@chrome://mochitests/content/browser/toolkit/components/contentrelevancy/tests/browser/browser_contentrelevancy_telemetry.js:57:37
[task 2024-04-26T02:54:33.465Z] 02:54:33     INFO - async*handleTask@chrome://mochikit/content/browser-test.js:1139:26
[task 2024-04-26T02:54:33.465Z] 02:54:33     INFO - _runTaskBasedTest@chrome://mochikit/content/browser-test.js:1211:18
[task 2024-04-26T02:54:33.465Z] 02:54:33     INFO - Tester_execTest@chrome://mochikit/content/browser-test.js:1353:14
[task 2024-04-26T02:54:33.465Z] 02:54:33     INFO - nextTest/<@chrome://mochikit/content/browser-test.js:1128:14
[task 2024-04-26T02:54:33.465Z] 02:54:33     INFO - SimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13
[task 2024-04-26T02:54:33.466Z] 02:54:33     INFO - Leaving test bound test_classify_succeed
[task 2024-04-26T02:54:33.466Z] 02:54:33     INFO - Entering test bound test_classify_fail_case1
[task 2024-04-26T02:55:18.633Z] 02:55:18     INFO - Not taking screenshot here: see the one that was previously logged
[task 2024-04-26T02:55:18.634Z] 02:55:18     INFO - TEST-UNEXPECTED-FAIL | toolkit/components/contentrelevancy/tests/browser/browser_contentrelevancy_telemetry.js | Test timed out - 
[task 2024-04-26T02:55:18.635Z] 02:55:18     INFO - GECKO(6032) | Completed ShutdownLeaks collections in process 6392

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/462f013c08ca
Add basic telemetry for content relevancy r=bdk

Comment 11

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/462f013c08ca