1858540 - Implement telemetry for enabling/disabling ads in review checker

Status: VERIFIED FIXED

(Firefox Graveyard :: Shopping, task, P2)

Description

[Ania]

To understand the ads disable rate, we would need the following probes:

shopping.surfaceAdsSettingToggled - event, recorded when user toggles the ads display settings.

shopping.settings.disabledAds - boolean property, indicates if the user has toggled off/disabled ads. Set during shopping component init and updated when changed in the sidebar.

Comment 1

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Attachment: Bug 1858540 - Implement telemetry events for enabling/disabling ads in review checker. r?perry.mcmanis

Comment 2

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Comment on attachment 9358402 [details]
Bug 1858540 - Implement telemetry events for enabling/disabling ads in review checker. r?perry.mcmanis

DATA REVIEW REQUEST

1. What questions will you answer with this data?

How often do users with ads enabled choose to disable the ads? And how many shopping sidebar users have disabled ads?

2. Why does Mozilla need to answer these questions? Are there benefits for users?
   Do we need this information to address product or business requirements?

Business wants to understand how users will respond to ads. Tracking ads being turned off is part of this.

3. What alternative methods did you consider to answer these questions?
   Why were they not sufficient?

There's no other way to track this data alongside the feature/user retention data we can gather via telemetry.

4. Can current instrumentation answer these questions?

No.

5. List all proposed measurements and indicate the category of data collection for each
   measurement, using the Firefox data collection categories found on the Mozilla wiki.

Measurement Name	Measurement Description	Data Collection Category	Tracking Bug
`shopping.surface_ads_setting_toggled#action	Whether the toggle was used to enable or disable ads. Possible values are enabled and disabled.	interaction	https://bugzilla.mozilla.org/show_bug.cgi?id=1858540
shopping_settings.disabled_ads	Indicates if the user has manually disabled ads. Set during shopping component init and updated when changed in browser.	interaction	https://bugzilla.mozilla.org/show_bug.cgi?id=1858540

6. Please provide a link to the documentation for this data collection which
   describes the ultimate data set in a public, complete, and accurate way.

This collection is Glean so is documented in the Glean Dictionary.

7. How long will this data be collected?

This collection has expiry '122'.

8. What populations will you measure?

All channels, countries, and locales. No filters.

9. If this data collection is default on, what is the opt-out mechanism for users?

These collections are Glean. The opt-out can be found in the product's preferences.

10. Please provide a general description of how you will analyze this data.

Data Science will create dashboards.

11. Where do you intend to share the results of your analysis?

Internally.

12. Is there a third-party tool (i.e. not Glean or Telemetry) that you
    are proposing to use for this data collection?

No.

Comment 3

[Rares Doghi, Desktop QA]

Hi @Jared can you explain these events a bit ?

The shopping.surface_ads_setting_toggled is displayed in Glean when the user switches this pref: browser.shopping.experience2023.ads.enabled in about:config ?

And this event shopping_settings.disabled_ads shows up in Glean when the user Turns it off from the Settings card ?

Comment 4

[Travis Long [:travis_]]

Comment on attachment 9358402 [details]
Bug 1858540 - Implement telemetry events for enabling/disabling ads in review checker. r?perry.mcmanis

Data Review

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, through the metrics.yaml file and the Glean Dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, through the data preferences in the application settings.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

N/A, collection to end or be renewed by version 122

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction data

5. Is the data collection request for default-on or default-off?

Default-on

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No

7. Is the data collection covered by the existing Firefox privacy notice?

Yes

8. Does the data collection use a third-party collection tool?

No

Result

data-review+

Comment 5

[:Gijs (he/him)]

(In reply to Rares Doghi, Desktop QA from comment #3)

Hi @Jared can you explain these events a bit ?

The shopping.surface_ads_setting_toggled is displayed in Glean when the user switches this pref: browser.shopping.experience2023.ads.enabled in about:config ?

This is an event, and it is recorded only when the setting is changed in the sidebar (not when changed in about:config).

And this event shopping_settings.disabled_ads shows up in Glean when the user Turns it off from the Settings card ?

This isn't an event, it's a boolean property that indicates the current state of ads (ie whether it's currently enabled or disabled). It should be have an explicit value irrespective of whether the user has recently toggled the setting in the sidebar or not.

Does that help?

Comment 6

[Rares Doghi, Desktop QA]

But we already have this pref browser.shopping.experience2023.ads.enabled which shows if the ads feature is turned on or off at a Firefox level and we already have the browser.shopping.experience2023.ads.userEnabled that shows if its enabled or disabled by the user from the shopping sidebar.

Is this pref: shopping_settings.disabled_ads the same as browser.shopping.experience2023.ads.userEnabled only the first one shows up in Glean ?
Would we have to submit a metrics type of ping in about:glean in order to see it in the Debug Ping viewer ?

Comment 7

[:Gijs (he/him)]

(In reply to Rares Doghi, Desktop QA from comment #6)

But we already have this pref browser.shopping.experience2023.ads.enabled which shows if the ads feature is turned on or off at a Firefox level and we already have the browser.shopping.experience2023.ads.userEnabled that shows if its enabled or disabled by the user from the shopping sidebar.

I'm confused - those are prefs. This bug is about probes (ie telemetry/glean data).

Is this pref: shopping_settings.disabled_ads the same as browser.shopping.experience2023.ads.userEnabled only the first one shows up in Glean ?

Yes. And the disabled_ads thingy should also show up as a scalar (shopping.disabled_ads) in about:telemetry, once the patch lands.

Would we have to submit a metrics type of ping in about:glean in order to see it in the Debug Ping viewer ?

I guess so but I don't know enough about glean, hopefully Perry can answer this.

Comment 8

[Rares Doghi, Desktop QA]

Thanks Gijs.

Comment 9

[Perry McManis [:perry.mcmanis]]

(In reply to Rares Doghi, Desktop QA from comment #6)

But we already have this pref browser.shopping.experience2023.ads.enabled which shows if the ads feature is turned on or off at a Firefox level and we already have the browser.shopping.experience2023.ads.userEnabled that shows if its enabled or disabled by the user from the shopping sidebar.

Is this pref: shopping_settings.disabled_ads the same as browser.shopping.experience2023.ads.userEnabled only the first one shows up in Glean ?
Would we have to submit a metrics type of ping in about:glean in order to see it in the Debug Ping viewer ?

That's correct, you will want to send a metrics ping via about:glean and these should show up.

Comment 10

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Moving open telemetry bugs into the general shopping backlog

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0e070e9c79b4
Implement telemetry events for enabling/disabling ads in review checker. r=TravisLong

Comment 12

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/0e070e9c79b4

Comment 13

[Rares Doghi, Desktop QA]

Verified as fixed in our latest Build 122.0a1 (2023-11-30) - 20231130172344.

Comment 14

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Comment on attachment 9358402 [details]
Bug 1858540 - Implement telemetry events for enabling/disabling ads in review checker. r?perry.mcmanis

Beta/Release Uplift Approval Request

• User impact if declined: For a future shopping experiment where we enable ads, product would like to measure how often users opt out of ads in the settings UI inside the shopping sidebar. This patch adds that telemetry.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is a telemetry patch to experimental code that is preffed off for all users except a very small fraction of users enrolled in shopping experiments. The patch itself is small and covered by automated tests.
• String changes made/needed: None
• Is Android affected?: No

Comment 15

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9358402 [details]
Bug 1858540 - Implement telemetry events for enabling/disabling ads in review checker. r?perry.mcmanis

Approved for 121.0b6.

Comment 16

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/c578799bcc6d

Comment 17

[Rares Doghi, Desktop QA]

Verified as fixed in our latest Beta Build 121.0b6.