Closed Bug 1756917 Opened 3 months ago Closed 3 months ago

Add telemetry for the top pick/best match checkbox in the preferences UI

Categories

(Firefox :: Address Bar, task, P1)

task
Points:
3

Tracking

()

VERIFIED FIXED
99 Branch
Tracking Status
firefox99 --- verified

People

(Reporter: adw, Assigned: daisuke)

References

(Blocks 1 open bug, )

Details

Attachments

(4 files)

Per the best match spec, we need telemetry for the following:

  • Top Pick preference toggles
  • Top Pick preferences learn more clicked

For the preference toggle, we can follow the example of the three other Suggest prefs. For each of these prefs, we have two probes:

  • An event recorded when the preference is changed:
  • The user's current value of the pref is recorded in telemetry environment:

For the learn more link, we probably want to record an event when it's clicked.

Rebecca, does that sound good?

Flags: needinfo?(rburwei)

yep, event telemetry for preference toggles and learn more clicks sound good to me.

Jeff, could we get the new Preference for Best Match added to clients_daily, and aggregated by choosing the final recorded value for the Preference on a given day?

Flags: needinfo?(rburwei) → needinfo?(jklukas)

(In reply to Rebecca BurWei from comment #3)

Jeff, could we get the new Preference for Best Match added to clients_daily

The pref name is: browser.urlbar.suggest.bestmatch

Attached file data-request.md

Data review request for three new probes related to the top pick feature of Firefox Suggest: two new events and one preference addition to telemetry environment data

Attachment #9265446 - Flags: data-review?(mmccorquodale)
Attachment #9265313 - Attachment description: Bug 1756917: Add telemetry for best match learm more link in the preferences page. → Bug 1756917: Add telemetry for best match learn more link in the preferences page.
See Also: → 1757353

Proposed change to clients_daily in https://github.com/mozilla/bigquery-etl/pull/2764 up for review.

Flags: needinfo?(jklukas)

Comment on attachment 9265446 [details]
data-request.md

Clearing data review request until we know what we're doing here.

Attachment #9265446 - Flags: data-review?(mmccorquodale)
Pushed by dakatsuka.birchill@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f005ae12bb44
Add telemetry for best match checkbox in the preferences page. r=adw
https://hg.mozilla.org/integration/autoland/rev/89b398ecfec3
Add telemetry for best match learn more link in the preferences page. r=preferences-reviewers,adw,Gijs

STR for QA:

  1. Enable Firefox Suggest and best match (browser.urlbar.bestMatch.enabled)
  2. Open about:preferences#privacy and scroll down to the Address Bar section
  3. Click the "Top pick" checkbox
  4. Open about:telemetry and search for browser.ui.interaction.preferences_panePrivacy
  5. Verify the Keyed Scalars section appears with browser.ui.interaction.preferences_panePrivacy under the parent process, and in the table under that, firefoxSuggestBestMatch appears with a Value of 1
  6. Go back to about:preferences and click the "Learn more" link next to the "Top pick" checkbox
  7. Go back to about:telemetry and refresh the page (search for browser.ui.interaction.preferences_panePrivacy again if necessary)
  8. In addition to firefoxSuggestBestMatch from step 5, verify that firefoxSuggestBestMatchLearnMore now appears in the table with a Value of 1
Flags: qe-verify+
Flags: in-testsuite+
Status: ASSIGNED → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → 99 Branch

I have verified this issue on the latest Nightly 99.0a1 (Build ID: 20220304094153) on Windows 10 x64, macOS 10.15.7 and Linux Ubuntu 20.04 x64.

  1. In order to verify this issue we have used the STR from comment 10.
  2. Each time the Top pick checkbox is clicked, the value of the firefoxSuggestBestMatch keyed scalar increases with 1.
  3. Each time the Learn more link is clicked, the value of the firefoxSuggestBestMatchLearnMore keyed scalar increases with 1.
  4. Both of the firefoxSuggestBestMatch and firefoxSuggestBestMatchLearnMore keyed scalars appear with browser.ui.interaction.preferences_panePrivacy under the parent process in the about:telemetry page.
Status: RESOLVED → VERIFIED
Flags: qe-verify+

(In reply to Alexandru Ionescu (needinfo me) [:alexandrui - PTO returning April 12] from comment #13)

(In reply to Pulsebot from comment #9)

Pushed by dakatsuka.birchill@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f005ae12bb44
Add telemetry for best match checkbox in the preferences page. r=adw
https://hg.mozilla.org/integration/autoland/rev/89b398ecfec3
Add telemetry for best match learn more link in the preferences page.
r=preferences-reviewers,adw,Gijs

== Change summary for alert #33467 (as of Sat, 05 Mar 2022 06:03:38 GMT) ==

Improvements:

Ratio Test Platform Options Absolute values (old vs new)
16% nytimes FirstVisualChange windows10-64-shippable-qr cold fission webrender 413.92 -> 347.42
13% nytimes loadtime windows10-64-shippable-qr cold fission webrender 1,793.02 -> 1,562.71
4% buzzfeed PerceptualSpeedIndex windows10-64-shippable-qr cold fission webrender 799.00 -> 764.92

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=33467

:alexandrui adding a telemetry probe is unlikely to cause page load improvements. Could you revisit the alert to see if there's a more likely cause?

Flags: needinfo?(aionescu)

done, thanks!

Flags: needinfo?(aionescu)
You need to log in before you can comment on or make changes to this bug.