1820273 - Add or confirm telemetry for onboarding-related interactions

Status: RESOLVED FIXED

(Fenix :: Experimentation and Telemetry, task, P1)

Description

[Joe M [:jmahon]]

The design team wants to be able to learn more about what features drive the highest levels of engagement in the existing onboarding flow. These are things they want to make sure are being tracked.

It's possible some or all of these are already tracked. If they're not, please implement the new probes. Some of them may also be expired or expiring soon - in that case, please renew them. And please reply to this bug with this table filled out appropriately:

event description	probe	new or existing?
"Collections placeholder" card on homepage	n/a	n/a
---> Dismissed
"Meet your personalized homepage" CFR on homepage	n/a	n/a
---> Dismissed with (x)
---> Dismissed passively
Pin private tab to home CFR	n/a	n/a
---> Dismissed with (x)
---> Dismissed passively
Set to default info card on homepage	n/a	n/a
---> Dismissed
---> Action button clicked
Add progressive app website to homepage CFR	n/a	n/a
---> Dismissed with (x)
---> Dismissed passively
"Customize homepage" button on homepage	n/a	n/a
---> Action button clicked

Notes:

• I threw this ticket together somewhat hastily. If I got one of the behaviors wrong, sorry - feel free to correct it.
• If you have questions, :emilyw would be a good person to start with.

Comment 1

[Joe M [:jmahon]]

[Tracking Requested - why for this release]:

Oh, I forgot to include this in the original description: if there are probe changes, they should be targeted for uplifts. This doesn't need to go straight to release, and it's not necessarily urgent, but the sooner we straighten out this telemetry, the sooner we can resolve some decisions about what to do in the associated Onboarding revamp project.

(Apologies to the RelMan team if I used any of these flags wrong - feel free to update the ticket or let me know if I messed anything up!)

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox111 (beta) and tracked for firefox112 (nightly). We have limited time to fix this, the soft freeze is in 3 days. However, the bug still isn't assigned.

:cpeterson, could you please find an assignee for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit auto_nag documentation.

Comment 3

[Joe M [:jmahon]]

[Tracking Requested - why for this release]: Apologies, when I marked this as tracking for v111 I didn't realize how soon soft freeze was. I'll leave it as ? for now (in the unlikely event that any of this telemetry does get merged in the next two days) but if that doesn't happen, it doesn't need to be rushed into v111.

Comment 4

[Joe M [:jmahon]]

Curious update: I was under the impression that 'passively' dismissing CFRs was intentional, but based on this old comment, it sounds like we're actually planning to prevent that from being possible. Bug 1812461 covers that change.

So, for the sake of this telemetry ticket, please document the existing probes for any of those "dismissed passively" items, but don't bother introducing any new probes for any that don't currently exist. We'll wind up removing the existing ones once that other bug is complete, anyway.

Comment 5

[Chris Peterson [:cpeterson]]

Joe, do we plan to run any onboarding experiments in v111 or v112? If not, do we still need to track and uplift these telemetry probes for v111 and v112?

Bugzilla's "Tracking" tracking flag (separate from the "Status" tracking flag) is usually reserved to flag Relman's attention to critical bugs like crash fixes or last-minute uplifts.

Comment 6

[Joe M [:jmahon]]

This isn't for experimentation, it's just for data science to be able to review existing behaviors. I didn't mean to convey critical urgency; feel free to update the tracking/status flags to whatever makes sense for "once this is done, it should be uplifted into beta, then release, as expediently as reasonable" and I'll try to follow that example for future use of the tracking flags.

Comment 7

[Donal Meehan [:dmeehan]]

[Tracking Requested - why for this release]:

Comment 8

[Chris Peterson [:cpeterson]]

(In reply to Joe M [:jmahon] from comment #6)

feel free to update the tracking/status flags to whatever makes sense for "once this is done, it should be uplifted into beta, then release, as expediently as reasonable" and I'll try to follow that example for future use of the tracking flags.

Donal updated the status-firefox111 flag to "affected" so we know that we'd like to uplift to a 111 dot release, so I think the flags are good now.

Comment 9

[Alexandru Putanu [:aputanu]]

Filled the table with existing probes:

Event description	Probe	New/Existing?
"Collections placeholder" card on homepage	N/A	N/A
->Dismissed	To be added in https://bugzilla.mozilla.org/show_bug.cgi?id=1821032	New
"Meet your personalized homepage" CFR on homepage	N/A	N/A
->Dismissed with X button	https://dictionary.telemetry.mozilla.org/apps/fenix/metrics/recent_tabs_jump_back_in_cfr_dismissed	Existing
->Dismissed passively	https://dictionary.telemetry.mozilla.org/apps/fenix/metrics/recent_tabs_jump_back_in_cfr_cancelled	Existing
Pin private tab to home CFR	N/A	N/A
---> Dismissed with “No thanks”	https://dictionary.telemetry.mozilla.org/apps/fenix/metrics/private_browsing_shortcut_cfr_cancel	Existing
---> Dismissed passively	None	N/A
Set to default info card on homepage	N/A	N/A
---> Dismissed	https://dictionary.telemetry.mozilla.org/apps/fenix/metrics/messaging_message_dismissed	Existing
---> Action button clicked	https://dictionary.telemetry.mozilla.org/apps/fenix/metrics/messaging_message_clicked	Existing
Add progressive app website to homepage CFR	N/A	N/A
---> Dismissed with "No thanks"	To be added in https://bugzilla.mozilla.org/show_bug.cgi?id=1821030	New
---> Dismissed passively	None	N/A
"Customize homepage" button on homepage	N/A	N/A
---> Action button clicked	https://dictionary.telemetry.mozilla.org/apps/fenix/metrics/home_screen_customize_home_clicked	Existing

Comment 10

[Joe M [:jmahon]]

This is fantastic, thanks Alexandru!

:emilyw, to summarize for your convenience:

• We're currently missing two probes:
  • The "Collections placeholder" card (on homepage) - bug 1821032.
  • The "progressive app website" CFR - bug 1821030.
• The items that do have telemetry are:
  • "Meet your personalized homepage" CFR
  • "Pin private tab to home" CFR
  • "Set to default" info card (on homepage)
  • "Customize homepage" button (on homepage)
• We'll be removing the ability to 'passively' dismiss CFRs in bug 1812461.

Other than the two missing ones, and the caveat about 'passive dismissal', is there anything else you would like to instrument in the short term? Additionally: how urgent are the two missing probes - should we be trying to update those ASAP?

(Reminder: we'll have the opportunity to come up with an appropriate instrumentation strategy for the overall onboarding flow as we build out that project)

Comment 11

[Emily Wachowiak [:emilyw]]

Thanks, Joe & Alexandru. I was able to add some more data to my FigJam flow here. We're using this data to help us evaluate which message touchpoints to keep and which to optimize. It's not the only deciding factor, though.

So given that, I don't think it's urgent to add in the missing telemetry since it would take 28 days post launch to get the data we need anyway. In general, I'd say it would be nice to have as much consistent data about onboarding as possible in the future since we'll always be wanting to optimize it (by consistent I mean that metrics like views, dismissals, and interactions are available for all components using the same structure/terms so it's easy to find them).

Comment 12

[Joe M [:jmahon]]

Cool, thanks, Emily. I think we can close this ticket, now - looks like Alex has already made progress on the two missing probes, which is great, but this means we don't need to schedule uplifts for them.

Thanks to both of you for the quick turnaround!