1555172 - Extend `setExperimentActive` to support enrollment IDs

Status: RESOLVED FIXED

(Toolkit :: Telemetry, enhancement, P2)

Description

[Michael Cooper [:mythmon]]

In order to help correlate enrollments, unenrollments and other events, and help distinguish between cloned profiles, Normandy would like to associate each call to setExperimentActive with a random, unique id that is generated when a user first enrolls in an experiment. This data should then be accessible by analysts in some way.

One way this could be done is extending the setExperimentActive method to allow for another option, enrollmentId. Normandy could then generate that data, persist it, and pass it to setExperimentActive when needed.

Comment 1

[Chris H-C :chutten]

Is this needed independently of (earlier than?) the broader project in bug 1549760?

Comment 2

[Michael Cooper [:mythmon]]

This is independent from 1549760. It's a separate request for more useful Telemetry from Normandy that happened to be at the same time as bug 1549760.

Comment 3

[Chris H-C :chutten]

Attachment: Bug 1555172 - Allow annotating experiments with Enrollment Ids. r?janerik!,mythmon!

This id, to be generated at runtime by the client, will be used to distinguish
when a client enrolls multiple times in the same id/branch/type.

Comment 4

[Chris H-C :chutten]

:mythmon, would you mind writing up the Data Collection Review for this? I'm not 100% clear on how this is intended to be used.

Alternatively, I suppose we could attach the Data Review to bug 1555178 where the IDs will actually be generated and submitted.

Comment 5

[Chris H-C :chutten]

Hey Sunah, does this look like a useful shape for this to be in for Data Engineering? Anything else needing to be done?

Comment 6

[bugzilla]

Looks good to me! I just filed a dependent bug for the data engineering side.

Comment 7

[Michael Cooper [:mythmon]]

Attachment: data-collection-review-1555172.md

Comment 8

[Teon Brooks [:teon] (ex-Mozilla)]

just need a bit of clarification:

• for the inclusion of the enrollment_id, will this be added as a scalar to the telemetry collection?
• and for the enrollment_id, this is an existing id, but will just be included in the environment?

Comment 9

[Michael Cooper [:mythmon]]

enrollment_id will be newly generated for each enrollment, and won't be re-used beyond a single experiment. I don't know how exactly this will be represented in the telemetry ping, for that I'd direct you towards :chutten.

We also plan to include the enrollment ID in other pings related to a users enrollment in an experiment, such as Telemetry Events.

Comment 10

[Chris H-C :chutten]

It'll be present in the Environment of every ping that contains an Environment. Documentation can be found here: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/data/environment.html

(( It'll be included in the example and explained in the experiments subsection. It will also have a section detailing its usage in the API documentation here: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/experiments.html#the-js-api ))

Comment 11

[Teon Brooks [:teon] (ex-Mozilla)]

Comment on attachment 9077814 [details]
data-collection-review-1555172.md

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, it will be present in the Environment of every ping that contains an Environment. Documentation can be found here: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/data/environment.html

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, :mythmon is responsible.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

Default on for all channels with Normandy enabled.

    Does the instrumentation include the addition of any new identifiers?

Yes, enrollment_id. The ID will expire after the experiment is completed.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/acd1ffd459f7
Allow annotating experiments with Enrollment Ids. r=janerik,mythmon

Comment 13

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/acd1ffd459f7

Comment 14

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Alicia, I noticed that this collection contains a new persistent ("life of an experiment") random identifier, and one possible parsing of the data collection review form is "escalate to legal if the data includes new identifiers"—I don't have any concerns but wanted to put your eyes on it. AIUI, Bug 1555176 will actually start reporting this identifier.

Comment 15

[Alicia Gray]

Hi Tim,

Thanks for the escalation. Trust & Security review = approved; Cat 2 interaction data. Bug 1555176 has also been updated accordingly with Trust signoff.