1555176 - Generate enrollment IDs and include them on all related telemetry events

Status: RESOLVED FIXED

(Firefox :: Normandy Client, enhancement)

Description

[Michael Cooper [:mythmon]]

When enrolling a user in an experiment (add-on or preference), an enrollment ID should be generated. For any subsequent event sent about that experiment, the enrollment ID should be included as an extra field. This will make it easier to analyze sequences of events for an experiment.

This will also help debug why we see users enrolling many times in an experiment, enrolling once but unenrolling many times, or other combinations of invalid events. The leading theory about why this happens is profile duplication, which causes many clients to share one Telemetry and Normandy ID. By randomly generated these IDs at the time of enrollment, we make it easier to identify cloned profiles.

Comment 1

[Su-Young Hong]

adding a requirement here, we need this enrollment ID to also be added to main pings. Reason being, usage metrics are calculated from main pings (or downstream from there, main summary, experiments table, search table, etc.) so having enrollment ID present there allows us to distinguish between cloned profiles sharing client_ids. If they're only present in event telemetry, we won't have this capability.

Comment 2

[Michael Cooper [:mythmon]]

That requirement is already captured in bug 1555178.

Comment 3

[Michael Cooper [:mythmon]]

Attachment: Bug 1555176 - Add enrollmentIds to all Normandy telemetry. r=Gijs

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f8a352e9f455
Add enrollmentIds to all Normandy telemetry. r=Gijs

Comment 5

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Friendly reminder that adding keys to telemetry events is a new collection so it should receive data review before landing in a build.

Super psyched for this!

Comment 6

[Michael Cooper [:mythmon]]

Attachment: bug-1555176-data-review.txt

This is very similar to the data review from bug 1555172, which introduced some of the APIs this feature uses (but didn't actually collect new data).

One thing to note is that current these identifiers are not cleared when Telemetry is disabled, which is a problem. I've filed bug 1584337 to cover that.

Comment 7

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Comment on attachment 9096400 [details]
bug-1555176-data-review.txt

This is a new identifier, so I'm redirecting to Alicia for Trust and Safety review.

Alicia, this is the same identifier described in bug 1555172, but being reported in additional probes.

==

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, in Events.yaml ("a unique ID for this enrollment that will be included in all related Telemetry").

2) Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available.

Yes, the [Firefox telemetry opt-out](https://support.mozilla.org/en-US/kb/share-data-mozilla-help-improve-firefox).

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, mythmon.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, technical data.

5) Is the data collection request for default-on or default-off?

Default-on.

6) Does the instrumentation include the addition of **any *new* identifiers**?

Yes.

7) Is the data collection covered by the existing Firefox privacy notice?

Escalating to legal since the data contains new identifiers.

8) Does there need to be a check-in in the future to determine whether to renew the data?

No, permanent collection.

9) Does the data collection use a third-party collection tool?

No.

Comment 8

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/f8a352e9f455

Comment 9

[Alicia Gray]

Trust & Security Data Collection Review:

Approved for the additional identifier. Note: I believe this is Cat 2 - interaction data, rather than Cat 1 - technical data, but if I'm misunderstanding how you are interpreting this ID, please advise.

• 1 to the requirement that if a user opts out of telemetry, this ID needs to be deleted. Adding Mark Reid for situational awareness as he works on telemetry deletion requirements for the California Consumer Privacy Act.

Comment 10

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Thanks, Alicia! I asserted that it's category 1 because if experiments are enabled, the enrollment IDs will be created automatically without any user intervention. The identifiers themselves are random and do not contain information. Knowing that an enrollment ID for a client exists doesn't let us infer anything about user actions (except that Firefox was running during the experiment's enrollment period, which is generally true for Cat 1 data).

Comment 11

[Alicia Gray]

(In reply to Tim Smith 👨‍🔬 [:tdsmith] from comment #10)

Thanks, Alicia! I asserted that it's category 1 because if experiments are enabled, the enrollment IDs will be created automatically without any user intervention. The identifiers themselves are random and do not contain information. Knowing that an enrollment ID for a client exists doesn't let us infer anything about user actions (except that Firefox was running during the experiment's enrollment period, which is generally true for Cat 1 data).

Thanks Tim. That makes sense. Bug 1555172 called it cat 2 so if there is a discrepancy in the way we define it for category purposes we should make sure we are clear why its Cat 2 in 1555172 and Cat 1 here. Looks like that is because this is the ID creation bug and 155172 is related to the events themselves.