Closed Bug 1412257 Opened 7 years ago Closed 7 years ago

track open/close/click events for onboarding telemetry

Categories

(Firefox :: Tours, enhancement, P3)

Product:
Firefox
Component:
Tours
Type:
enhancement

Tracking

()

Status:
RESOLVED DUPLICATE of bug 1413830

People

(Reporter: gasolin, Assigned: gasolin)

References

Details

(Whiteboard: [onboarding-telemetry])

Attachments

(1 file)

Bug 1412257 - track open/close/click events for onboarding telemetry;
59 bytes, text/x-review-board-request
Details 
overlay-nav-click now triggered when go through the goto function

We could only trigger the event when actual click in navigation menu happens
Assignee: nobody → gasolin
Status: NEW → ASSIGNED
also add `onboarding_icon_click` event here so we don't have to cross-reference sessions table to find the icon click number
Comment on attachment 8923277 [details]
Bug 1412257 - track open/close/click events for onboarding telemetry;

https://reviewboard.mozilla.org/r/194458/#review199432

::: commit-message-d3910:1
(Diff revision 2)
> +Bug 1412257 - More accurate overlay-nav-click event for onboarding telemetry;r=fischer

Please update the commit message and the bug title if you intends to do 2 pings in one bug.

::: browser/extensions/onboarding/content/onboarding.js:541
(Diff revision 2)
>        case "onboarding-overlay-button":
>          this.showOverlay();
> -        this.gotoPage(this._firstUncompleteTour.id);
> +        let target_tour_id = this._firstUncompleteTour.id;
> +        this.gotoPage(target_tour_id);
> +        telemetry({
> +          event: "onboarding_icon_click",

In the table Product provided, looks like they expects a different event name. I have no preference about the event name just to make sure every body is aligned. Please talk to Product for the desired name.
Attachment #8923277 - Flags: review?(fliu)
I think the Event that Product provided (`Click-Firefox-Logo`/`Click-Overlay-CTA`) are just a reference and help us get the idea of what they want from the telemetry. Only the result is important and it does not mean we need to use exactly the same name in telemetry.

(ex: we use `overlay-cta-click` instead of `Click-Overlay-CTA` now)

To align with existing events name I prefer `onboarding-logo-click` or `onboarding-icon-click` to identify the event.

Cindy, do you have any concern on the naming part?
Flags: needinfo?(chsiang)
Comment on attachment 8923277 [details]
Bug 1412257 - track open/close/click events for onboarding telemetry;

https://reviewboard.mozilla.org/r/194458/#review199440

::: commit-message-d3910:1
(Diff revision 2)
> +Bug 1412257 - More accurate overlay-nav-click event for onboarding telemetry;r=fischer

Since we have some concern on the the new event naming, I'll move `onboarding-logo-click` event out of this patch

::: browser/extensions/onboarding/content/onboarding.js:541
(Diff revision 2)
>        case "onboarding-overlay-button":
>          this.showOverlay();
> -        this.gotoPage(this._firstUncompleteTour.id);
> +        let target_tour_id = this._firstUncompleteTour.id;
> +        this.gotoPage(target_tour_id);
> +        telemetry({
> +          event: "onboarding_icon_click",

I'll remove this to make the patch scope clear
Please assign priorities when filing new bugs that you're going to work on in the Tours component.
Priority: -- → P3
(In reply to Matthew N. [:MattN] (PM if requests are blocking you) from comment #8)
> Please assign priorities when filing new bugs that you're going to work on
> in the Tours component.

Thanks for reminding, will do at next time
After discussion, we also want to add some new events. New events will be added in this bug or in the following bug, here's the data review request.


1) What questions will you answer with this data?

To analysis user interaction with onboarding tool for better browser retention.

2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements?

We need this information to address product requirements. Data will be used for pursuing better browser retention rate.

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?

We can use current overlay-nav-click and overlay-session event, but `overlay-nav-click` can't identify the user actually clicks the nav button or the user just see the tour because the tour showed to them by default.
overlay-session event can identify user open/close the overlay but can't identify how the user opens/close the overlay.

4) Can current instrumentation answer these questions?

No

5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox [data c](https://wiki.mozilla.org/Firefox/Data_Collection)[ollection ](https://wiki.mozilla.org/Firefox/Data_Collection)[categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the found on the Mozilla wiki.   

<table>
  <tr>
    <td>Measurement Description</td>
    <td>Data Collection Category</td>
    <td>Tracking Bug #</td>
  </tr>
  <tr>
    <td>onboarding-logo-click triggered when user open the overlay via click the logo or the watermark</td>
    <td>Interaction data</td>
    <td>Bug 1412257</td>
  </tr>
  <tr>
    <td>overlay-show-tour event is sent when any tour is displayed on the overlay</td>
    <td>Interaction data</td>
    <td>Bug 1412257</td>
  </tr>
  <tr>
    <td>overlay-close-button-click triggered when user close the overlay via click the overlay close button</td>
    <td>Interaction data</td>
    <td>Bug 1412257</td>
  </tr>
  <tr>
    <td>overlay-close-outside-click triggered when user close the overlay via click outside of the tour content area</td>
    <td>Interaction data</td>
    <td>Bug 1412257</td>
  </tr>
</table>



6) How long will this data be collected?  Choose one of the following:

* I want to permanently monitor this data. (Fred Lin)

7) What populations will you measure?

* Which release channels? All

* Which countries? All

* Which locales? All

* Any other filters?  Please describe in detail below.
No

8) Please provide a general description of how you will analyze this data.

Identify how user open/close the overlay and click the tour

9) Where do you intend to share the results of your analysis?

The data will be shared between PM & UX to drive the better decision
Flags: needinfo?(liuche)
Whiteboard: [onboarding-telemetry]
Comment on attachment 8923277 [details]
Bug 1412257 - track open/close/click events for onboarding telemetry;

https://reviewboard.mozilla.org/r/194458/#review200340

::: browser/extensions/onboarding/content/onboarding.js:581
(Diff revision 3)
>        // Keep focus (not visible) on current item for potential keyboard
>        // navigation.
>        target.focus();
> +      telemetry({
> +        event: "overlay-nav-click",
> +        tour_id: tourId,

Where is `tourId` declared?
Attachment #8923277 - Flags: review?(fliu)
I will rate this as at least P2 since we keep and keep collecting the incorrect data
Priority: P3 → P2
newly add open/close/click events are similar, so I'd plan to add them in this bug

The patch use `overlay-current-tour` instead of `overlay-show-tour` now. The wording might change but they should track same events.
Summary: More accurate overlay-nav-click event for onboarding telemetry → track open/close/click events for onboarding telemetry
update data review request accordingly

1) What questions will you answer with this data?

To analysis user interaction with onboarding tool for better browser retention.

2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements?

We need this information to address product requirements. Data will be used for pursuing better browser retention rate.

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?

We can use current overlay-nav-click and overlay-session event, but `overlay-nav-click` can't identify the user actually clicks the nav button or the user just see the tour because the tour showed to them by default.
overlay-session event can identify user open/close the overlay but can't identify how the user opens/close the overlay.

4) Can current instrumentation answer these questions?

No

5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox [data c](https://wiki.mozilla.org/Firefox/Data_Collection)[ollection ](https://wiki.mozilla.org/Firefox/Data_Collection)[categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the found on the Mozilla wiki.   

<table>
  <tr>
    <td>Measurement Description</td>
    <td>Data Collection Category</td>
    <td>Tracking Bug #</td>
  </tr>
  <tr>
    <td>onboarding-logo-click triggered when user open the overlay via click the logo or the watermark</td>
    <td>Interaction data</td>
    <td>Bug 1412257</td>
  </tr>
  <tr>
    <td>overlay-current-tour event is sent when any tour is going to display on the overlay</td>
    <td>Interaction data</td>
    <td>Bug 1412257</td>
  </tr>
  <tr>
    <td>overlay-close-button-click triggered when user close the overlay via click the overlay close button</td>
    <td>Interaction data</td>
    <td>Bug 1412257</td>
  </tr>
  <tr>
    <td>overlay-close-outside-click triggered when user close the overlay via click outside of the tour content area</td>
    <td>Interaction data</td>
    <td>Bug 1412257</td>
  </tr>
</table>



6) How long will this data be collected?  Choose one of the following:

* I want to permanently monitor this data. (Fred Lin)

7) What populations will you measure?

* Which release channels? All

* Which countries? All

* Which locales? All

* Any other filters?  Please describe in detail below.
No

8) Please provide a general description of how you will analyze this data.

Identify how user open/close the overlay and click the tour

9) Where do you intend to share the results of your analysis?

The data will be shared between PM & UX to drive the better decision
Comment on attachment 8923277 [details]
Bug 1412257 - track open/close/click events for onboarding telemetry;

https://reviewboard.mozilla.org/r/194458/#review200340

> Where is `tourId` declared?

fixed
Comment on attachment 8923277 [details]
Bug 1412257 - track open/close/click events for onboarding telemetry;

https://reviewboard.mozilla.org/r/194458/#review200432

Since this bug's scope has been broaden to some new pings, would like to wait for the discussion with Ptoduct on Thur then proceed.
Attachment #8923277 - Flags: review?(fliu)
(In reply to Fischer [:Fischer] from comment #12)
> I will rate this as at least P2 since we keep and keep collecting the
> incorrect data

Fred, for previous data that is being collected, if those probes are incorrect or not useful, can you file a bug to remove them from collection?

Thanks for the data document! It looks like there will still be discussion about how to break this bug down, so please flag me for data review when that's settled, unless you think the data collection and pings will not change?
Flags: needinfo?(liuche)
Comment on attachment 8923277 [details]
Bug 1412257 - track open/close/click events for onboarding telemetry;

Will wait for the results of Thursday meeting, because it sounds like the data pings collected may change based on that discussion.
Attachment #8923277 - Flags: review?(liuche)
(In reply to Fischer [:Fischer] from comment #12)
> I will rate this as at least P2 since we keep and keep collecting the
> incorrect data

The current event can fulfill PM's original purpose (to count the engage rate by this tour's cta-button-click / this tour's total impression)

So we have more time to think what's the extra data need to be collected
Priority: P2 → P3
Will do the change in bug 1413830 (with new table)
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Flags: needinfo?(chsiang)
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: