1828234 - Collect blocked advertisers in Glean New Tab metrics

Status: VERIFIED FIXED

(Firefox :: New Tab Page, task, P1)

Description

[Lina Butler [:lina]]

This ticket covers adding a new metric for blocked advertisers on the New Tab page. We'll use this data to guide our inventory and fill projections.

Blocked advertisers are stored in the browser.topsites.blockedSponsors pref, as a JSON array of strings. Glean has a string_list metric type that lets us set the value to an array directly (docs here: https://mozilla.github.io/glean/book/reference/metrics/string_list.html#set).

We also have a function, _beginObservingNewtabPingPrefs, that records metrics from prefs at startup, and adds an observer to re-record them whenever any of the prefs change: https://searchfox.org/mozilla-central/rev/7939a5150dcd96915bccf1c819433ad489a5edc9/browser/components/newtab/lib/TelemetryFeed.jsm#1296-1330 Let's extend this to also support the blockedSponsors pref.

Comment 1

[Tif Tran]

Attachment: request.md

Comment 2

[Tif Tran]

Attachment: Bug 1828234 - Collect blocked advertisers in Glean New Tab metrics r=lina

Comment 3

[Lina Butler [:lina]]

Thanks Tif! Would you mind making a couple of tweaks to your data review request, please? 😊

    <td> browser.topsites.blockedSponsors</td>
    <td> List of Sponsored Tiles that have been dimissed</td>
    <td>3</td>

1. The measurement name is newtab.blocked_sponsors. browser.topsites.blockedSponsors is the name of the pref.
2. Let's be a bit more precise for the description, and say something like "list of advertiser names for Sponsored Tiles that have been dismissed". It'll be correct in the Glean Dictionary, anyway, but...
3. This is category 2 (interaction), not 3 (web_activity).

Comment 4

[Tif Tran]

Comment on attachment 9330303 [details]
request.md

># Request for data collection review form
>
>**All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.**
>
>1) What questions will you answer with this data?
>    
>    We will have context of which sponsored tiles were dismissed.
>2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:
>
>*  This will be used to guide our inventory and fill projections
>
>3) What alternative methods did you consider to answer these questions? Why were they not sufficient?
>
>    Without blocked sponsors we won't know what tiles have been blocked
>
>4) Can current instrumentation answer these questions?
>
>    No, since it hasn't been collected by Glean
>
>5) List all proposed measurements and indicate the category of data collection for each measurement, using the [Firefox data collection categories](https://wiki.mozilla.org/Data_Collection) found on the Mozilla wiki.   
>
>**Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.**
>
><table>
>  <tr>
>    <td>Measurement Name</td>
>    <td>Measurement Description</td>
>    <td>Data Collection Category</td>
>    <td>Tracking Bug #</td>
>  </tr>
>  <tr>
>    <td> newtab.blocked_sponsors</td>
>    <td> list of advertiser names for Sponsored Tiles that have been dismissed </td>
>    <td>2</td>
>    <td>https://bugzilla.mozilla.org/show_bug.cgi?id=1828234</td>
>  </tr>
></table>
>
>6) Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.
> 
> * This collection is documented in the Glean Dictionary at https://dictionary.telemetry.mozilla.org/
>
>
>7) How long will this data be collected?  Choose one of the following:
>
>* The NewTab working group will permanently monitor this data. 
>
>8) What populations will you measure?
>
>* We will measure all channels, countries and locales.
>
>9) If this data collection is default on, what is the opt-out mechanism for users?
>
>* These collections are Glean. The opt-out can be found in the product's preferences.
>
>10) Please provide a general description of how you will analyze this data.
>
>* This data will be used to guid inventory and fill projections
>
>11) Where do you intend to share the results of your analysis?
>
>* The results would be collected into a looker dashboard
>
>12) Is there a third-party tool (i.e. not Glean or Telemetry) that you are proposing to use for this data collection? If so:
>
>* No

Comment 5

[Lina Butler [:lina]]

Great, thank you!

In the future, could you please upload a new attachment and "obsolete" the old one, instead of using "Edit Attachment As Comment"? (It's totally okay that you did it this time, Bugzilla's attachments UI can be a little clunky if you aren't used to it! 😊)

1. Click "Attach New File", upload or paste the fixed-up form into that field, and set its description (it doesn't have to have the same one as your current one, but it can) and "data-review: ?" flag, like you would for a brand-new review request.
2. In the "Obsoletes" section, just underneath all the flags, check the box next to the old request.md.
3. (Optionally: Check the box next to "Clear the needinfo request for <your email>", if it isn't checked already, just above the Submit button. You can also do that later in the main bug view—on this page underneath the comment box).
4. Click Submit!

Bugzilla will upload your new form, and hide your old request.md (it'll still be there, but references to it will have a strikethrough) and clear any flags on it.

Comment 6

[Megan McCorquodale]

Comment on attachment 9330303 [details]
request.md

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
   Yes, this will be documented in the Glean dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?
   Yes, users can opt out of telemetry collection.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?
   Yes, the New Tab working group will monitor.

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
   Category 2, Interaction Data.

5. Is the data collection request for default-on or default-off?
   Default on.

6. Does the instrumentation include the addition of any new identifiers?
   No new identifiers.

7. Is the data collection covered by the existing Firefox privacy notice?
   Yes.

8. Does the data collection use a third-party collection tool?
   No.

---

data-review +

Comment 7

[Tif Tran]

[Tracking Requested - why for this release]: This telemetry is required for revenue projections of sponsored tiles in Fx 113.

Comment 8

[Tif Tran]

Steps to Reproduce:

• Open browser with sponsored tiles enabled
• Open a new tab and go to about:glean
• Under "About Testing" 2) select drop down to "New Tab"
• Click the "Apply settings and submit ping"
• Make note of the value in the textbox in point 1, go to https://debug-ping-preview.firebaseapp.com/ and find the entry
• Ensure a there is an existing payload has "newtab.blocked_sponsors" listed
• Go to the newtab page and dismiss one or two sponsored tiles
• Go back the about:glean page and click the button again
• Go to the debug-ping-preview page and see whether a new payload is produced with the "newtab.blocked_sponsors".

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/88aec4fa33c1
Collect blocked advertisers in Glean New Tab metrics r=lina

Comment 10

[Ryan VanderMeulen [:RyanVM]]

We've already shipped our final 113 beta of the cycle and are building the RC on Monday. If this needs to ship in 113, we'll need a Beta approval request on the patch ASAP.

Comment 11

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/88aec4fa33c1

Comment 12

[Ryan VanderMeulen [:RyanVM]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #10)

Comment 13

[Lina Butler [:lina]]

Comment on attachment 9330304 [details]
Bug 1828234 - Collect blocked advertisers in Glean New Tab metrics r=lina

Beta/Release Uplift Approval Request

• User impact if declined: Collecting telemetry for dismissed sponsored tiles on the New Tab page one release cycle earlier will help Mozilla with revenue projections.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce: If manual testing is desired, given how close we are to branching:

1. Open the New Tab page (about:newtab).
2. Click the "..." button next to a "Sponsored" tile, and choose "Dismiss" from the menu.
3. In a separate tab, open about:glean.
4. In about:glean, select "newtab" from the drop-down list in step 2 of the "About Testing" section. (Optionally, change the default "debug tag" in step 1, though this isn't necessary).
5. Click "Apply settings and submit ping".
6. Open the Glean Debug Ping Viewer link referenced in step 5 of the "About Testing" section.
7. After a few seconds, verify in the Glean Debug Ping Viewer that a new ping is submitted with the tag from Step 4, and that its Payload contains the key "newtab.blocked_sponsors", with a list containing the name of the advertiser from the dismissed tile from Step 2.
8. Repeat Steps 2-7 for a different sponsored tile, and verify that the latest submitted ping contains both advertisers from the two dismissed tiles.

• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is a small patch with no dependencies that adds a new Glean telemetry probe, and a pref observer to update the probe when the list of dismissed tiles changes. It has automated test coverage, and can be verified manually in Nightly using the Glean Debug Ping Viewer.
• String changes made/needed: None.
• Is Android affected?: No

Comment 14

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9330304 [details]
Bug 1828234 - Collect blocked advertisers in Glean New Tab metrics r=lina

Approved for 113.0rc1, thanks.

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/c49389f57a85

Comment 16

[Cosmin Muntean [:cmuntean], Ecosystem QA]

I have verified this issue on the latest Nightly 114.0a1 build (Build ID: 20230507095340) and the latest Beta 113.0 (Build ID: 20230504192738) on Windows 10 x64, macOS 13.1 and Linux Mint 20.

• The blocked Sponsored Tiles are correctly displayed in glean. Here is an example:

  "string_list": {
  "newtab.blocked_sponsors": [
  "amazon",
  "expedia"
  ]
  }