Closed Bug 1412164 Opened 7 years ago Closed 7 years ago

add new tour_type and timestamp column for onboarding telemetry

Categories

(Firefox :: Tours, enhancement, P3)

Product:
Firefox
Component:
Tours
Type:
enhancement

Tracking

()

Status:
RESOLVED FIXED
Milestone:
Firefox 58
Tracking Flags:
Tracking Status
firefox57 --- fixed
firefox58 --- fixed

People

(Reporter: gasolin, Assigned: gasolin)

References

Details

(Whiteboard: [onboarding-telemetry])

Attachments

(1 file)

Bug 1412164 - add tour_type and timestamp column for onboarding telemetry;
59 bytes, text/x-review-board-request
liuche
: review+
Fischer
: review+
ritu
: approval-mozilla-beta+
Details 
After doing SQL query with redash https://sql.telemetry.mozilla.org/dashboard/onboarding we'd like add some fields to able to identify new/update user

The plan is 

* add `tour_type` ("new" or "update") column for both session and event ping. (so we can identify the action is bind to new or update user)
* add `tour_source` column to event ping (so we can identify the event is happen when the icon is in firefox or waterfall state)


This bug is related to the client side work, will file a server side change bug in ping-centre repo.


Confirmed with Najiang, we can parallel the client/server side change.
"the server will populate some default value for the new columns. So you don't have to wait anything to land the client code
the etl jobs on the server will just ignore those new columns until we add them, it wont break anything there."
server side bug is filed in https://github.com/mozilla/ping-centre/issues/121
Assignee: nobody → gasolin
Status: NEW → ASSIGNED
Comment on attachment 8922599 [details]
Bug 1412164 - add tour_type and timestamp column for onboarding telemetry;

https://reviewboard.mozilla.org/r/193714/#review198864

::: commit-message-aa958:1
(Diff revision 1)
> +Bug 1412164 - add new column for onboarding telemetry;r=fischer,liuche

Yesterday, Product asked me questions about the pings. We probably should have a brief discussion with Product before proceeding. Would you help this?
Attachment #8922599 - Flags: review?(fliu)
After discussion we also like to add `timestamp` column for events (Timestamp in (integer) milliseconds when the event triggered) so we can calculate the drop rate by comparing session begin/end timestamp and the event time timestamp.

Other further changes could be filed as the followup.


Cindy, could you confirm that?
Flags: needinfo?(chsiang)
That's correct. 
Fred, kindly file separate bugs for 
-FF icon click event (with icon state and speech bubble state)
- Navigation button click

Thanks.
Flags: needinfo?(chsiang)
Thanks Cindy.

ChenXia, could you help on the data review part?
Flags: needinfo?(liuche)
Comment on attachment 8922599 [details]
Bug 1412164 - add tour_type and timestamp column for onboarding telemetry;

https://reviewboard.mozilla.org/r/193714/#review199390

Per the last meeting with Product, the current overlay-nav-click is measuring the wrong ping. I believe the work to correct thar just requires moving few lines of codes. Why not fixing this? I can foresee if not fixing this in the eariler stage, we will receieve thousands of wrong pings. Could we know the reason of not doing that after your discussion with Product.
Attachment #8922599 - Flags: review?(fliu)
Comment on attachment 8922599 [details]
Bug 1412164 - add tour_type and timestamp column for onboarding telemetry;

> Per the last meeting with Product, the current overlay-nav-click is measuring the wrong ping. 

This patch is about adding new columns, and we want to make sure this critical patch could be land smoothly.

This doesn't mean we won't make overlay-nav-click event more accurate at time, but I prefer to send PR on a separate Bug 1412257, that part does not need data review and we can request uplift there.

So please kindly review again
Attachment #8922599 - Flags: review?(fliu)
Comment on attachment 8922599 [details]
Bug 1412164 - add tour_type and timestamp column for onboarding telemetry;

https://reviewboard.mozilla.org/r/193714/#review199436

::: browser/extensions/onboarding/OnboardingTelemetry.jsm:142
(Diff revision 2)
>      } = this.state.sessions[session_key];
>      let category = EVENT_WHITELIST[event].category;
> +    // the field is used to identify how user open the overlay (through default logo or watermark),
> +    // the number of open from notification can be retrieved via `notification-cta-click` event
> +    let tour_source = Services.prefs.getStringPref("browser.onboarding.state", "default");
> +    let tour_type = Services.prefs.getStringPref("browser.onboarding.tour-type", "update");

Since `tour_type` is a must and sent on every ping could we make one of registration param like `page`?
Attachment #8922599 - Flags: review?(fliu)
Comment on attachment 8922599 [details]
Bug 1412164 - add tour_type and timestamp column for onboarding telemetry;

https://reviewboard.mozilla.org/r/193714/#review199436

> Since `tour_type` is a must and sent on every ping could we make one of registration param like `page`?

good idea, will do it
Comment on attachment 8922599 [details]
Bug 1412164 - add tour_type and timestamp column for onboarding telemetry;

https://reviewboard.mozilla.org/r/193714/#review199484

r+ with required issue fixed

::: browser/extensions/onboarding/content/onboarding.js:404
(Diff revision 4)
>      this._resizeTimerId =
>        this._window.requestIdleCallback(() => this._resizeUI());
>      registerNewTelemetrySession({
>        page: this._window.location.href,
>        session_key: this._session_key,
> +      tour_type: Services.prefs.getStringPref("browser.onboarding.tour-type", "update"),

please reuse `this._tourType`
Attachment #8922599 - Flags: review?(fliu) → review+
Comment on attachment 8922599 [details]
Bug 1412164 - add tour_type and timestamp column for onboarding telemetry;

https://reviewboard.mozilla.org/r/193714/#review199486

::: commit-message-aa958:1
(Diff revision 4)
> +Bug 1412164 - add new column for onboarding telemetry;r=fischer,liuche

Please say "add tour_type" column
Summary: add new column for onboarding telemetry → add new tour_type column for onboarding telemetry
Comment on attachment 8922599 [details]
Bug 1412164 - add tour_type and timestamp column for onboarding telemetry;

https://reviewboard.mozilla.org/r/193714/#review199486

> Please say "add tour_type" column

also `timestamp` column don't forget it
Summary: add new tour_type column for onboarding telemetry → add new tour_type and timestamp column for onboarding telemetry
Comment on attachment 8922599 [details]
Bug 1412164 - add tour_type and timestamp column for onboarding telemetry;

https://reviewboard.mozilla.org/r/193714/#review199484

> please reuse `this._tourType`

updated
Comment on attachment 8922599 [details]
Bug 1412164 - add tour_type and timestamp column for onboarding telemetry;

https://reviewboard.mozilla.org/r/193714/#review199486

> also `timestamp` column don't forget it

updated
Hi Fred, we're adding a new process for Data Collection, so please fill out this document as part of data review. The goal is to simplify the process so that every time you make a data request, if you have this filled out to demonstrate the reasoning behind data review *and* have clear in-tree documentation, you'll get an r+ from any data steward and don't need any back and forth!

https://github.com/mozilla/data-review

and specifically this page: https://github.com/mozilla/data-review/blob/master/request.md

You can just copy paste it here as a comment.

Thanks! This is still a new process so please let me know if you have any feedback. I know this is a little more work right now, but the goal is to move to a process where data review is clear.
Flags: needinfo?(liuche) → needinfo?(gasolin)
Please assign priorities when filing new bugs that you're going to work on in the Tours component.
Priority: -- → P3
Thanks for point out the data review form, here is the request.


1) What questions will you answer with this data?

To identify the new/update user and analysis their interaction with onboarding tool for better browser retention.

2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

We need this information to address product requirements. Data will be used for pursuing better browser retention rate.

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?

we can guess some tour is only shown to new user/update user, but currently there's no way to clearly identify the user is a new user or the update user when we met the tour that show for both type of user.

4) Can current instrumentation answer these questions?

No

5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox [data c](https://wiki.mozilla.org/Firefox/Data_Collection)[ollection ](https://wiki.mozilla.org/Firefox/Data_Collection)[categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the found on the Mozilla wiki.   

<table>
  <tr>
    <td>Measurement Description</td>
    <td>Data Collection Category</td>
    <td>Tracking Bug #</td>
  </tr>
  <tr>
    <td>tour type is for identify the user is a `new` user or the `update` user (who upgrade from the older version)</td>
    <td>Interaction data</td>
    <td>Bug 1412164</td>
  </tr>
  <tr>
    <td>timestamp is for record when the event is happend so data analysis can compare table to find some insight in user sessions</td>
    <td>Interaction data</td>
    <td>Bug 1412164</td>
  </tr>
</table>



6) How long will this data be collected?  Choose one of the following:

* I want to permanently monitor this data. (Fred Lin)

7) What populations will you measure?

* Which release channels? All

* Which countries? All

* Which locales? All

* Any other filters?  Please describe in detail below.
No

8) Please provide a general description of how you will analyze this data.

New user retention, Update user retention, Further analyze events within a session

9) Where do you intend to share the results of your analysis?

The data will be shared between PM & UX to drive the better decision
Flags: needinfo?(gasolin) → needinfo?(liuche)
Whiteboard: [onboarding-telemetry]
Comment on attachment 8922599 [details]
Bug 1412164 - add tour_type and timestamp column for onboarding telemetry;

https://reviewboard.mozilla.org/r/193714/#review200602

data-review only: r+

> Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate? (see here, here, and here for examples). Refer to the appendix for "documentation" if more detail about documentation standards is needed.
    
Yes, the documentation is in data_events.md

> Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available.

Yes, this is under Firefox Telemetry


> If the request is for permanent data collection, is there someone who will monitor the data over time?

Permanent, Fred Gasolin

> Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Type 2

> Is the data collection request for default-on or default-off?

Default on

> Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No

> Is the data collection covered by the existing Firefox privacy notice? 

Yes

> Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)

no

::: browser/extensions/onboarding/data_events.md:78
(Diff revision 5)
>  | `locale` | The browser chrome's language (eg. en-US). | :two:
>  | `page` | [Required] One of ["about:newtab", "about:home"]| :one:
>  | `session_begin` | Timestamp in (integer) milliseconds when onboarding/overlay/notification becoming visible. | :one:
>  | `session_end` | Timestamp in (integer) milliseconds when onboarding/overlay/notification losing focus. | :one:
>  | `session_id` | [Required] The unique identifier generated by `gUUIDGenerator` service to identify the specific user session when onboarding is inited/when overlay is opened/when notification is shown. | :one:
> -| `tour_source` | Either ("default", "watermark") indicates the overlay is opened while in the default or the watermark state. Open from the notification bar is counted via 'notification-cta-click event'. | :two:
> +| `timestamp` | Timestamp in (integer) milliseconds when the event triggered | :one:

Thank you for adding the units (milliseconds) :)
Comment on attachment 8922599 [details]
Bug 1412164 - add tour_type and timestamp column for onboarding telemetry;

https://reviewboard.mozilla.org/r/193714/#review200606
Attachment #8922599 - Flags: review?(liuche) → review+
Flags: needinfo?(liuche)
Pushed by flin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/12a1cc903ec7
add tour_type and timestamp column for onboarding telemetry;r=Fischer,liuche
https://hg.mozilla.org/mozilla-central/rev/12a1cc903ec7
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
status-firefox58: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 58
Comment on attachment 8922599 [details]
Bug 1412164 - add tour_type and timestamp column for onboarding telemetry;

Approval Request Comment
[Feature/Bug causing the regression]: N
[User impact if declined]: PM can't identify the onboarding event is from a new or update user
[Is this code covered by automated tests?]: N, but will fail existing tests if telemetry broken
[Has the fix been verified in Nightly?]: N, manual local test works fine
[Needs manual test from QE? If yes, steps to reproduce]: N 
[List of other uplifts needed for the feature/fix]: N
[Is the change risky?]: N
[Why is the change risky/not risky?]: only add new telemetry column, no extra user interaction events
[String changes made/needed]: N
Attachment #8922599 - Flags: approval-mozilla-beta?
Comment on attachment 8922599 [details]
Bug 1412164 - add tour_type and timestamp column for onboarding telemetry;

Onboarding tour team needs this change in 57, Beta57+
Attachment #8922599 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: