Report Normandy user_id in telemetry environment
Categories
(Firefox :: Normandy Client, enhancement)
Tracking
()
| Tracking | Status | |
|---|---|---|
| firefox68 | --- | affected |
People
(Reporter: tdsmith, Unassigned)
Details
Normandy uses a user_id, which is a randomly generated UUID which is distinct from the client_id [1], for experiment targeting.
If clients in the wild have wound up with matching client_ids but diverging user_ids as a result of some cloning mishap, that means that producing metrics for users enrolled in an experiment by joining a list of enrolled client_ids onto e.g. main_summary could include extra rows from users that would never have enrolled, even if they met all of the other criteria for the experiment (hardware, locale, country, etc).
To figure out whether or not this is a real problem, the easiest course of action is probably to start collecting Normandy user_ids in telemetry and see how many distinct user_ids we have for each client_id.
Throwing app.normandy.user_id into this list (and filing for data steward review) would be the quickest way to get started.
If this ever happens, we can use it as a join key; if it never happens, we can stop collecting.
|
Reporter | |
Comment 1•7 years ago
|
||
A reason not to do this is if the Normandy ID is more persistent than the client ID or has different reset semantics, which I think it might.
|
||
Comment 2•7 years ago
|
||
A reason not to do this is if the Normandy ID is more persistent than the client ID or has different reset semantics, which I think it might.
That's correct. The Normandy ID currently does not ever get reset automatically.
|
|
Reporter | |
Comment 3•7 years ago
|
||
This seems like a non-starter.
|
|
Reporter | |
Comment 4•7 years ago
|
||
We confirmed that this is a real problem in https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/135684/command/135685 — you see something like 40% excess activity if you rely on client_id without considering the tag status of each individual ping, which will dilute the effect of any experiment. There are other mitigations planned for this, like bug 1555176 and bug 1555161.
Description
•