1723860 - Add event telemetry for Firefox Suggest onboarding modal prompt

Status: VERIFIED FIXED

(Firefox :: Address Bar, task, P1)

Description

[Drew Willcoxon :adw]

Rebecca requested we add telemetry to each button in the Firefox Suggest onboarding prompt. Event telemetry would be fine.

However, Natalie says we're going to change the prompt content, so this is blocked on that.

Also, we should let Rebecca know the names of the probes once we implement this because she needs to file a ticket with Data Engineering to make sure the data shows up in their backend.

Comment 1

[Drew Willcoxon :adw]

[Tracking Requested - why for this release]: This is required for the Firefox Suggest feature on 92.

Comment 2

[Rebecca BurWei]

Drew & Dao - I added more details about the telemetry requirements here: https://mozilla-hub.atlassian.net/browse/SNT-30

Please let me know if you need more clarification.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Moving tracking the metabug for this work.

Comment 4

[Dão Gottwald [:dao]]

Attachment: Bug 1723860 - Add event telemetry for Firefox Suggest onboarding modal prompt. r=adw

Comment 5

[Drew Willcoxon :adw]

Attachment: request.md

Comment 6

[Teon Brooks [:teon]]

Comment on attachment 9240649 [details]
request.md

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions file Events.yaml and the Probe Dictionary.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, :adw is responsible.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---

Result: datareview+

Comment 7

[Drew Willcoxon :adw]

https://treeherder.mozilla.org/jobs?repo=try&revision=bb0bfb5073706a283c4647620eed8f92c1a8921d

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fbc127829141
Add event telemetry for Firefox Suggest onboarding modal prompt. r=dao

Comment 9

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/fbc127829141

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

Since the status are different for nightly and release, what's the status for beta?
For more information, please visit auto_nag documentation.

Comment 11

[Drew Willcoxon :adw]

STR for QA

1. First you'll need to trigger the online onboarding dialog. You could start with the online scenario using Nimbus, but it would probably be easier set the relevant prefs since there are several things to verify in the dialog and it goes away each time you click one. So, set the following prefs and restart:

browser.urlbar.quicksuggest.enabled=true
browser.urlbar.quicksuggest.shouldShowOnboardingDialog=true
browser.urlbar.suggest.quicksuggest=false
browser.urlbar.suggest.quicksuggest.sponsored=false

Also delete browser.urlbar.quicksuggest.showedOnboardingDialog if it exists, which it will after the dialog has been shown.

2. Click the "Allow suggestions" button in the dialog
3. Open about:telemetry#events-tab and verify the following events:

category=contextservices.quicksuggest method=enable_toggled    object=enabled
category=contextservices.quicksuggest method=sponsored_toggled object=enabled
category=contextservices.quicksuggest method=opt_in_dialog     object=accept

4. Re-enable the dialog as in step 1 and restart
5. Click the "Customize in settings" button in the dialog
6. Open about:telemetry#events-tab and verify the following event:

category=contextservices.quicksuggest method=opt_in_dialog object=settings

7. Re-enable the dialog as in step 1 and restart
8. Click the "Not now" link in the dialog
9. Open about:telemetry#events-tab and verify the following event:

category=contextservices.quicksuggest method=opt_in_dialog object=not_now

10. Re-enable the dialog as in step 1 and restart
11. Click the "Learn more" link in the dialog (it's in the text in the middle)
12. Open about:telemetry#events-tab and verify the following event:

category=contextservices.quicksuggest method=opt_in_dialog object=learn_more

Comment 12

[Drew Willcoxon :adw]

Attachment: 93/mozilla-beta & 92/mozilla-release patch

Approval Request Comment
[Feature/Bug causing the regression]: Firefox Suggest offline/online rollouts
[User impact if declined]: Needed for important rollouts on 93 and 92
[Is this code covered by automated tests?]: Yes
[Has the fix been verified in Nightly?]: No
[Needs manual test from QE? If yes, steps to reproduce]: Yes, please see comment 11
[List of other uplifts needed for the feature/fix]: See uplift coordination spreadsheet
[Is the change risky?]: Low risk relative to other uplifts needed in the patch stack
[Why is the change risky/not risky?]: This patch adds new telemetry events that are recorded when users click the various buttons in the Firefox Suggest online onboarding opt-in dialog. No visible changes.
[String changes made/needed]:

Comment 13

[Drew Willcoxon :adw]

Comment on attachment 9240774 [details] [diff] [review]
93/mozilla-beta & 92/mozilla-release patch

Beta/Release Uplift Approval Request

• User impact if declined: Needed for important rollouts on 93 and 92
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Yes, please see comment 11
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This patch adds new telemetry events that are recorded when users click the various buttons in the Firefox Suggest online onboarding opt-in dialog. No visible changes.
• String changes made/needed:

Comment 14

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9240774 [details] [diff] [review]
93/mozilla-beta & 92/mozilla-release patch

Needed for experiments targeting Fx92. Approved for 93.0b4.

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/7865281de699

Comment 16

[Carmen Fat [:cfat] - Ecosystem QA]

• I have verified this issue using the latest Nightly 94.0a1 (Build ID: 20210912213037) and Beta 93.0b4 (Build ID: 20210912185727) builds on Windows 10 x64, macOS 11.5.2, and Linux Mint 20.

• I have verified that after allowing suggestions from the modal, the following events are triggered:
  category=contextservices.quicksuggest method=enable_toggled object=enabled category=contextservices.quicksuggest method=sponsored_toggled object=enabled category=contextservices.quicksuggest method=opt_in_dialog object=accept

• I have verified that after clicking the "Customize in settings" button from the modal, the following event is triggered:
  category=contextservices.quicksuggest method=opt_in_dialog object=settings

• I have verified that after clicking the "Not now" button from the modal, the following event is triggered:
  category=contextservices.quicksuggest method=opt_in_dialog object=not_now

• I have verified that after clicking the "Learn more" button from the modal, the following event is triggered:
  category=contextservices.quicksuggest method=opt_in_dialog object=learn_more

All the events mentioned above were verified using a local Telemetry server I previously setup on the machine to make sure that the probes are reaching their destination.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9240774 [details] [diff] [review]
93/mozilla-beta & 92/mozilla-release patch

Approved for 92.0.1.

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/60596434f5ce

Comment 19

[Carmen Fat [:cfat] - Ecosystem QA]

• I have verified this issue using the Candidate 92.0.1 build (Build ID: 20210920202613) on Windows 10 x64, macOS 10.15.7 and Linux Ubuntu 20.04 x64.

• I have verified that after allowing suggestions from the modal, the following events are triggered:
  category=contextservices.quicksuggest method=enable_toggled object=enabled category=contextservices.quicksuggest method=sponsored_toggled object=enabled category=contextservices.quicksuggest method=opt_in_dialog object=accept

• I have verified that after clicking the "Customize in settings" button from the modal, the following event is triggered:
  category=contextservices.quicksuggest method=opt_in_dialog object=settings

• I have verified that after clicking the "Not now" button from the modal, the following event is triggered:
  category=contextservices.quicksuggest method=opt_in_dialog object=not_now

• I have verified that after clicking the "Learn more" button from the modal, the following event is triggered:
  category=contextservices.quicksuggest method=opt_in_dialog object=learn_more