1756917 - Add telemetry for the top pick/best match checkbox in the preferences UI

Status: VERIFIED FIXED

(Firefox :: Address Bar, task, P1)

Description

[Drew Willcoxon :adw]

Per the best match spec, we need telemetry for the following:

• Top Pick preference toggles
• Top Pick preferences learn more clicked

For the preference toggle, we can follow the example of the three other Suggest prefs. For each of these prefs, we have two probes:

• An event recorded when the preference is changed:
  • Recorded here
  • Defined here
  • Tested here
  • Documented here
• The user's current value of the pref is recorded in telemetry environment:
  • Recorded here
  • Tested here (same as above)
  • Documented here

For the learn more link, we probably want to record an event when it's clicked.

Rebecca, does that sound good?

Comment 1

[Daisuke Akatsuka (:daisuke)]

Attachment: Bug 1756917: Add telemetry for best match checkbox in the preferences page.

Comment 2

[Daisuke Akatsuka (:daisuke)]

Attachment: Bug 1756917: Add telemetry for best match learn more link in the preferences page.

Depends on D139575

Comment 3

[Rebecca BurWei]

yep, event telemetry for preference toggles and learn more clicks sound good to me.

Jeff, could we get the new Preference for Best Match added to clients_daily, and aggregated by choosing the final recorded value for the Preference on a given day?

Comment 4

[Drew Willcoxon :adw]

(In reply to Rebecca BurWei from comment #3)

Jeff, could we get the new Preference for Best Match added to clients_daily

The pref name is: browser.urlbar.suggest.bestmatch

Comment 5

[Drew Willcoxon :adw]

Attachment: data-request.md

Data review request for three new probes related to the top pick feature of Firefox Suggest: two new events and one preference addition to telemetry environment data

Comment 6

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/bigquery-etl/pull/2764

Comment 7

[Jeff Klukas [:klukas] (UTC-4)]

Proposed change to clients_daily in https://github.com/mozilla/bigquery-etl/pull/2764 up for review.

Comment 8

[Drew Willcoxon :adw]

Comment on attachment 9265446 [details]
data-request.md

Clearing data review request until we know what we're doing here.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f005ae12bb44
Add telemetry for best match checkbox in the preferences page. r=adw
https://hg.mozilla.org/integration/autoland/rev/89b398ecfec3
Add telemetry for best match learn more link in the preferences page. r=preferences-reviewers,adw,Gijs

Comment 10

[Drew Willcoxon :adw]

STR for QA:

1. Enable Firefox Suggest and best match (browser.urlbar.bestMatch.enabled)
2. Open about:preferences#privacy and scroll down to the Address Bar section
3. Click the "Top pick" checkbox
4. Open about:telemetry and search for browser.ui.interaction.preferences_panePrivacy
5. Verify the Keyed Scalars section appears with browser.ui.interaction.preferences_panePrivacy under the parent process, and in the table under that, firefoxSuggestBestMatch appears with a Value of 1
6. Go back to about:preferences and click the "Learn more" link next to the "Top pick" checkbox
7. Go back to about:telemetry and refresh the page (search for browser.ui.interaction.preferences_panePrivacy again if necessary)
8. In addition to firefoxSuggestBestMatch from step 5, verify that firefoxSuggestBestMatchLearnMore now appears in the table with a Value of 1

Comment 11

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/f005ae12bb44
https://hg.mozilla.org/mozilla-central/rev/89b398ecfec3

Comment 12

[Carmen Fat [:cfat] - Ecosystem QA]

I have verified this issue on the latest Nightly 99.0a1 (Build ID: 20220304094153) on Windows 10 x64, macOS 10.15.7 and Linux Ubuntu 20.04 x64.

1. In order to verify this issue we have used the STR from comment 10.
2. Each time the Top pick checkbox is clicked, the value of the firefoxSuggestBestMatch keyed scalar increases with 1.
3. Each time the Learn more link is clicked, the value of the firefoxSuggestBestMatchLearnMore keyed scalar increases with 1.
4. Both of the firefoxSuggestBestMatch and firefoxSuggestBestMatchLearnMore keyed scalars appear with browser.ui.interaction.preferences_panePrivacy under the parent process in the about:telemetry page.

Comment 13

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Pulsebot from comment #9)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f005ae12bb44
Add telemetry for best match checkbox in the preferences page. r=adw
https://hg.mozilla.org/integration/autoland/rev/89b398ecfec3
Add telemetry for best match learn more link in the preferences page.
r=preferences-reviewers,adw,Gijs

== Change summary for alert #33467 (as of Sat, 05 Mar 2022 06:03:38 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
16%	nytimes FirstVisualChange	windows10-64-shippable-qr	cold fission webrender	413.92 -> 347.42
13%	nytimes loadtime	windows10-64-shippable-qr	cold fission webrender	1,793.02 -> 1,562.71
4%	buzzfeed PerceptualSpeedIndex	windows10-64-shippable-qr	cold fission webrender	799.00 -> 764.92

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=33467

Comment 14

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Alexandru Ionescu (needinfo me) [:alexandrui - PTO returning April 12] from comment #13)

(In reply to Pulsebot from comment #9)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f005ae12bb44
Add telemetry for best match checkbox in the preferences page. r=adw
https://hg.mozilla.org/integration/autoland/rev/89b398ecfec3
Add telemetry for best match learn more link in the preferences page.
r=preferences-reviewers,adw,Gijs

== Change summary for alert #33467 (as of Sat, 05 Mar 2022 06:03:38 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
16%	nytimes FirstVisualChange	windows10-64-shippable-qr	cold fission webrender	413.92 -> 347.42
13%	nytimes loadtime	windows10-64-shippable-qr	cold fission webrender	1,793.02 -> 1,562.71
4%	buzzfeed PerceptualSpeedIndex	windows10-64-shippable-qr	cold fission webrender	799.00 -> 764.92

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=33467

:alexandrui adding a telemetry probe is unlikely to cause page load improvements. Could you revisit the alert to see if there's a more likely cause?

Comment 15

[Alexandru Ionescu (needinfo me) [:alexandrui]]

done, thanks!