1388648 - [meta] [Onboarding] [Telemetry] Add telemetry probe and build dashboard for the Onboarding tour

Status: RESOLVED FIXED

(Firefox :: General, enhancement, P1)

Description

[chsiang]

This is the meta bug to track onboarding telemetry and dashboard using ping center.

Comment 1

[[e-mail]]

After discuss with Activity Stream team, we'll focus on collect onboarding notification and overlay ping events within onboarding addon at first stage. Then explore other possibility when ping-centre module is integrated into firefox.

Current discussion and break down are cooked in google doc
https://docs.google.com/a/mozilla.com/document/d/1hc6RLoWglQ58Uq4n9ofjMLhwTSWQcs7q9K4AAmjk-ow/edit?usp=sharing

Will file related bugs when we have more clear sight on this.

Comment 2

[[e-mail]]

Based on the prototype board
https://sql.telemetry.mozilla.org/dashboard/onboarding

Here is the current check result

* Onboarding Daily Sessions
Onboarding session by date
  - Should only count event=onboarding_session

* Onboarding Daily Events
number of users interacting with onboarding
  - Should calc none repeated client id

* Onboarding_Notification Engagement
  - current have Total engagement, can add New, Update number

* Overlay Engagement by Tour
  - description: Performance, Default Browser, Sync CTA count are auto generate
  - current have Total engagement, can add New, Update number

* Onboarding Interaction
  - Add overlay-close-button-click & overlay-close-outside-click events once available

Per tour interaction
  - Calc drop rate by (total Tour1 impression/total Tour1 cta click)

Comment 3

[[e-mail]]

for `drop rate` I mean `daily engage rate` if it make sense

Comment 4

[[e-mail]]

As discussed on slack, redash does not allow co-edit, so I'll provide unpublished query so you can integrate them on onboarding dashboard and do changes anytime you want.

Updated: onboarding sessions (only count how many tabs created)
https://sql.telemetry.mozilla.org/queries/48510

Updated: onboarding daily users that interact with any onboarding elements (overlay/notification)  (with right user number)
https://sql.telemetry.mozilla.org/queries/48511

New: daily users that opened the overlay  (with right user number)
https://sql.telemetry.mozilla.org/queries/48518

Update: engagement daily users that click the overlay or notification cta button (with right user number)
https://sql.telemetry.mozilla.org/queries/48519

Comment 5

[Fischer [:Fischer]]

At this stage, adding the telemetry pings should be P1.

Comment 6

[Nicole Yee (:nicoleyee)]

Hey Fischer,

Unfortunately, if there are any other bugs that have not yet landed, we cannot uplift them to 57. The last beta went to build yesterday. I worked with Fred to get Bug 1412164 out the door and he said that was the highest priority for telemetry. All others will have to wait till 58.

Comment 7

[[e-mail]]

As discussed in last Friday with Cindy & engineers, we won't trace new telemetry columns (except Bug 1412164) for v57. 
So the priority is not related to v57 but just related to our current tasks priority.

Comment 8

[[e-mail]]

new/update user can be queried by adding `tour_type = 'new' or 'tour_type = update'` clause

Comment 9

[[e-mail]]

New/Update Users who open tour overlay by day
https://sql.telemetry.mozilla.org/queries/48942/source#132220

Comment 10

[[e-mail]]

Onboarding overlay that filtered out the experiment events
https://sql.telemetry.mozilla.org/queries/49127#132287

Comment 11

[[e-mail]]

how many home/newtabs created daily (replace Onboarding Daily Sessions)
https://sql.telemetry.mozilla.org/queries/49128#132291

Comment 12

[[e-mail]]

Engagement with Firefox logo when it’s active (colored) vs. when it’s timed out (watermark)
(filter overlay-session + tour_source = ‘default’/tour_source = ‘watermark’)

https://sql.telemetry.mozilla.org/queries/49129#132294

Comment 13

[[e-mail]]

Daily user > onboarding user https://sql.telemetry.mozilla.org/queries/49174#132400

Can use to replace New Tab User by day_57 Release & Onboarding Tour User by day_57 Release

Comment 14

[[e-mail]]

Onboarding overlay engagement https://sql.telemetry.mozilla.org/queries/49175#table

Can use to replace `Overlay Engagement by Tour`

Comment 15

[[e-mail]]

After discussion, Cindy decides we are ok with the current dashboard state.

We will build new dashboard and send fx-metrics review with new telemetry data in bug 1419995