1896004 - New Tab Wallpapers: Add Glean Telemetry

Status: VERIFIED FIXED

(Firefox :: New Tab Page, defect)

Description

[Nathan Barrett [:nbarrett]]

Add telemetry items for the following events:
Add telemetry for the following items:

• Clicks on wallpaper
• Change Wallpaper after setting it
• turning off wallpaper

Comment 1

[Nathan Barrett [:nbarrett]]

Attachment: Bug 1896004 - Add wallpaper telemetry r=thecount

Comment 2

[Nathan Barrett [:nbarrett]]

Comment on attachment 9401220 [details]
Bug 1896004 - Add wallpaper telemetry r=thecount

HNT Wallpaper feature- Data Review Questionnaire

Request for data collection review form

All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.

1. What questions will you answer with this data?

• How do users interact with the new feature
• Does the new feature increase engagement with home + new tab page?
• Once they have a wallpaper, are they changing wallpapers again or removing them.

2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

**Yes, we want to make sure that this new feature is something that users are interacting with - Personalization and customization are goals for HNT, and wallpaper is the first step toward that. In addition it would be helpful to know if users who view the wallpapers are also interacting with the customization controls **

3. What alternative methods did you consider to answer these questions? Why were they not sufficient? Since this is a new feature we wouldn’t have any data points related to wallpapers.

4. Can current instrumentation answer these questions? No.

5. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.

<table>
<tr>
<td>Measurement Description
</td>
<td>Data Collection Category
</td>
<td>Tracking Bug #
</td>
</tr>
<tr>
<td>newtab.wallpaper_click -
<p>
Is triggered when a user clicks on any of the wallpapers which would update the background image of the newtab page
</td>
<td>interaction
</td>
<td>1895797
</td>
</tr>
</table>

6. Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.

This collection is documented in the Glean Dictionary at https://dictionary.telemetry.mozilla.org/

7. How long will this data be collected? Choose one of the following

I want to permanently monitor this data. (Nathan Barrett)

8. What populations will you measure?

Most use cases are going to be experiments limited to en-US, but we still want this data for all users.

9. If this data collection is default on, what is the opt-out mechanism for users?

You can opt out of data collection in prefs, about:preferences#privacy, or turn off newtab.

10. Please provide a general description of how you will analyze this data.

Experimentation, comparing branches with control.

11. Where do you intend to share the results of your analysis?

The analysis will remain internal, and used by Newtab teams to improve the experience for users.

12. Is there a third-party tool (i.e. not Glean or Telemetry) that you are proposing to use for this data collection? If so:

No

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5770f86f73a1
Add wallpaper telemetry r=thecount,home-newtab-reviewers

Comment 4

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/5770f86f73a1

Comment 5

[Scott [:thecount] Downe]

[Tracking Requested - why for this release]: For beta and for the planned dot release that goes out on May 28th.

We wanted to get this out into an experiment sooner than 128 or 127 if possible.

These changes are adding in 1 click event to the wallpaper personalize menu, so when a user clicks a wallpaper thumbnail in the personalize menu, we send a click event to telemetry feed and then telemetry feed triggers a glean record. We also defined the event in glean metrics.yaml.

Comment 6

[Scott [:thecount] Downe]

[Tracking Requested - why for this release]: As I mentioned in the beta tracking comment, we are hoping this can be included in the planned dot release going out May 28th.

Comment 7

[Donal Meehan [:dmeehan]]

Tracking as requested, but the beta/release uplift will depend on getting an uplift request in time and the risk mentioned.
:nbarret can you add uplift requests on this when ready?

Comment 8

[Scott [:thecount] Downe]

Comment on attachment 9401220 [details]
Bug 1896004 - Add wallpaper telemetry r=thecount

Beta/Release Uplift Approval Request

• User impact if declined: This impacts the wallpaper experiment timeline.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Given Glean requires an ENV set at Firefox build time, I am not aware of a way to verify the telemetry.

So I think all we can do is do regression testing mainly around setting, clearing, and resetting wallpapers.

We need to rely on developers to verify the telemetry itself.

• List of other uplifts needed: None
• Risk to taking this patch: Medium
• Why is the change risky/not risky? (and alternatives if risky): This one has a bit of layers in my decision.

The code changes are medium, but the logic is simple, it's a lot of boiler plate.

Most of the changes are behind a pref.

There is also the QA bit that makes it harder to verify.

The biggest risk here is the telemetry doesn't work, very little risk of this breaking something unrelated to the telemetry, which we don't have today anyway.

So good regression testing for overall wallpapers and newtab should reduce 99% of this risk.

• String changes made/needed: None
• Is Android affected?: No

Comment 9

[Ciprian Muresan [:cmuresan], Ecosystem QA]

Hey Scott, while trying to verify the telemetry event I found the following:

• I get a TypeError: can't access property "record", Glean.newtab.wallpaperClick is undefined error in the browser console whenever I change the new tab wallpaper
• I haven't been able to find the newtab.wallpaper_click ping after tagging all events or newtab telemetry pings with a custom debug tag
• the probe dictionary site is reporting that the metric is no longer defined in the source code
  • The description of the ping also needs an update, it looks like it's the same as the one for the newtab.search ping
• I haven't been able to find a single event in looker in the last 7 days. I was expecting to see at least one.
• I've also set up a local telemetry server and I still could not generate one ping containing the newtab.wallpaper_click event.

I've used Firefox Nightly Version 128.0a1, Build ID 20240521213834 to perform the check. @scott, or @nbarret could either of you take a look at this?

Comment 10

[Pascal Chevrel:pascalc]

Why was the data-review set to minus?

Comment 11

[Nathan Barrett [:nbarrett]]

Data review was set to minus because while I initially filled out a data review, I realized that it was data-classification-low and didn't need a data-review with the new policy

Comment 12

[Nathan Barrett [:nbarrett]]

:cmuresan - will take a look today

Comment 13

[Donal Meehan [:dmeehan]]

Removing tracking and setting 126 to wontfix.
The request has a medium risk.

Comment 14

[Scott [:thecount] Downe]

Ciprian, looks like bug 1895797 broke something for this bug in nightly.

We're fixing it in bug 1898287, and this is for nightly.

In beta bug 1895797 does not exist, so the regression should not impact uplifting this bug to beta.

So I think this bug should still merge cleanly to beta without any bugs.

Comment 15

[Scott [:thecount] Downe]

Hey Pascal, just clarifying something I mentioned in the last comment, and trying to be super clear.

The patch in this bug made it into nightly, and another bug, bug 1895797 broke something.

However this patch should still be viable for beta as bug 1895797 is not in beta.

Comment 16

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/7c87ae2278f6

Comment 17

[Pascal Chevrel:pascalc]

Comment on attachment 9401220 [details]
Bug 1896004 - Add wallpaper telemetry r=thecount

Approved for 127 beta 6, thanks.

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1895797

Comment 19

[Maxx Crawford [:maxx]]

Attachment: Bug 1896004 - Add wallpaper telemetry r=thecount

Comment 20

[Phabricator Automation]

Comment on attachment 9403623 [details]
Bug 1896004 - Add wallpaper telemetry r=thecount

Revision D211296 was moved to bug 1895797. Setting attachment 9403623 [details] to obsolete.

Comment 21

[Ciprian Muresan [:cmuresan], Ecosystem QA]

I have verified that the telemetry ping is successfully sent to the server using a Firefox Nightly 128.0a1 Build ID 20240523205926 and a Firefox Beta 127.0b6 Build ID 20240523092204 on Windows 10, macOS 14, and Ubuntu 22.04. For each selected wallpaper a wallpaper_click event is successfully recorded. It contains extra information regarding the selected wallpaper, whether the user previously had a wallpaper, a new tab visit ID, and a glean timestamp.