Send telemetry trailhead enrollment ping

VERIFIED FIXED in Firefox 67

Status

()

enhancement
P1
normal
VERIFIED FIXED
4 months ago
2 months ago

People

(Reporter: Mardak, Assigned: nanj)

Tracking

(Blocks 1 bug, {github-merged})

unspecified
Firefox 68
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox67 verified, firefox68 verified)

Details

Attachments

(3 attachments)

moved to bug 1549846 For survey support, sounds like simplest approach would be to set a string pref with the combined branches similar to the trailhead.firstrun.branches but only if the user is in an experiment. (branches is an override that prevents experimentation and is used for nightly default and testing)

For enrollment ping, we should do this once probably when setting the above pref the first time. And do enrollment for both interrupts and triplets experiments

This survey is scoped to experiment 1 which has 5 branches: control, join, sync, nofirstrun, cards. We'll set a string pref to the branch name (no triplet) for normandy to target.

For enrollment, we'll want to record a telemetry event:

https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html#recordevent

normandy calls as so:
https://searchfox.org/mozilla-central/rev/99a2a5a955960b0e58ceade1db1f7652d9db4ba1/toolkit/components/normandy/lib/PreferenceExperiments.jsm#380
recordEvent("normandy", "enroll", "preference_study", name, {experimentType, branch});

We'll need new "activity-stream" category (the first argument) and that means we'll need to register a new event dynamically or in Events.yaml but pretty similar to the normandy definition:
https://searchfox.org/mozilla-central/rev/99a2a5a955960b0e58ceade1db1f7652d9db4ba1/toolkit/components/telemetry/Events.yaml#246-262

felix, do you have suggestions for how the enrollment ping should look?

Flags: needinfo?(flawrence)

This event has enough data for my needs (the experiment name, branch, a label for me to query and implicitly the client_id). You should check with a data platform engineer whether this form of event requires any work on their end (e.g. whether there's validation that this might conflict with), and if so whether there's an alternative that requires no work on their part.

Flags: needinfo?(flawrence)

Splitting out survey string pref setting from telemetry stuff

Assignee: edilee → najiang
Summary: Update experimentation behavior to support surveys and enrollment ping → Send telemetry trailhead enrollment ping
Depends on: 1549846

(In reply to Felix Lawrence from comment #2)

This event has enough data for my needs (the experiment name, branch, a label for me to query and implicitly the client_id). You should check with a data platform engineer whether this form of event requires any work on their end (e.g. whether there's validation that this might conflict with), and if so whether there's an alternative that requires no work on their part.

Felix - Thanks for the input!

We've already had an event called "activity-stream" in Events.yaml, I believe we can simply add a new entry "enroll" under it, and then request the data review for it.

No longer depends on: 1549846

Register a new telemetry event for Trailhead enrollment.

:chutten - could you take a look at this please? This records a ping in the Events data pipeline when a user gets enrolled in a Trailhead experiment. You can see more details in the attached request form.

This patch only modified the Events.yaml, and I haven't documented this ping in the Activity Stream yet. Will add it in when I implement the client side code.

Attachment #9063501 - Flags: data-review?(chutten)
Iteration: --- → 68.4 - Apr 29 - May 12

PR for the client side code is up, please see the telemetry document here

:chutten, could you take a data r? now?

Thanks!

Flags: needinfo?(chutten)
Comment on attachment 9063501 [details]
data_review_request_trailhead_enrollment.txt

1) Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes: https://github.com/mozilla/activity-stream/pull/5002/files#diff-f9d20d6688eef2754298456e5d5b836b

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the user can opt-out the data collection by either disabling the telemetry of Activity Stream or disabling the Firefox telemetry as a whole.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

Nan Jiang (najiang@mozill.com) and Ed Lee (edilee@mozilla.com) will be monitoring those metrics.

4) Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2

5) Is the data collection request for default-on or default-off?

Default-on.

6) Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

No.

9) Does the data collection use a third-party collection tool?

No, they are using Mozilla Events telemetry pipeline for this.
Attachment #9063501 - Flags: data-review?(chutten) → data-review+
Pushed by najiang@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c67422ffde6e
Register a telemetry event for trailhead enrollment r=Mardak,chutten

Can users really opt-out of the data collection? Doesn't this ping sent before they get a chance to opt out?

Blocks: 1550300

(In reply to Felix Lawrence from comment #11)

Can users really opt-out of the data collection? Doesn't this ping sent before they get a chance to opt out?

Good point. Practically, the enrolled users would have a fairly tight window to opt out, i.e. they can only opt out before the recorded ping gets sent to the backend, which then depends on how often Events telemetry pings to the backend.

chutten, what's your take on this? Do we need to take another look at this?

Keywords: github-merged
Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 68
Blocks: 1550098

(In reply to Nan Jiang [:nanj] from comment #12)

(In reply to Felix Lawrence from comment #11)

Can users really opt-out of the data collection? Doesn't this ping sent before they get a chance to opt out?

Good point. Practically, the enrolled users would have a fairly tight window to opt out, i.e. they can only opt out before the recorded ping gets sent to the backend, which then depends on how often Events telemetry pings to the backend.

chutten, what's your take on this? Do we need to take another look at this?

Is this collection Telemetry? If so it is subject to the datareporting policy mechanism on first start where no Telemetry is sent for 30min, giving the user a chance to read the Privacy Notice (opened for them in a tab) and find and disable the setting if they so choose.

To my knowledge, even if the collection isn't Telemetry, it still needs to adhere to this restriction. It would need to implement it itself.

Flags: needinfo?(chutten) → needinfo?(najiang)

Is this collection Telemetry? If so it is subject to the datareporting policy mechanism on first start where no Telemetry is sent for 30min, giving the user a chance to read the Privacy Notice (opened for them in a tab) and find and disable the setting if they so choose.

Yes, I think this is Telemetry. It records an event (only once) when a user first opens the browser through either a fresh install or a new profile, and gets enrolled in a Trailhead experiment. And this would happen prior to the Privacy Notice page gets shown to that user.

A few questions:

  • This is in fact a special Normandy preference study, which conducted without using its experimentation framework. Would Normandy's data collection policy also apply to here?
  • Like Normandy, we use Events telemetry pipeline to collect this metric, does Events telemetry client already support the datareporting policy (i.e. 30 min silence time on first start)?

The reporting policy applies to data upload, not data recording. So if there's data being recorded early, it's fine. Data being reported/uploaded/sent/otherwise leaving Firefox would not be fine.

All Telemetry adheres to this control, so if it's going through a Telemetry.* API you are compliant.

Yes, it's going through Services.telemetry.*.

Thanks for the clarification!

Flags: needinfo?(najiang)

I have verified that after setting "trailhead.firstrun.branches" to an empty string, setting "trailhead.firstrun.didSeeAboutWelcome" to false, restarting and navigating to about:welcome, an enrollment ping is sent and is visible in about:telemetry#events-tab on Nightly 68.0a1 (Build ID 20190512214232) on Windows 10 x64, macOS 10.14, and Arch Linux 4.14.3.

Status: RESOLVED → VERIFIED

Marking verified as per bug 1550098 Comment 13 for bugs that were status-firefox68=verified

Blocks: 1552285
Component: Activity Streams: Newtab → Messaging System
You need to log in before you can comment on or make changes to this bug.