[meta] Send historical experiments information over Telemetry
Categories
(Toolkit :: Telemetry, enhancement, P3)
Tracking
()
People
(Reporter: bugzilla, Unassigned)
References
Details
(Keywords: meta)
This has been discussed a bit in the past as a desirable thing but in particular for ease of analysis of clients that are prematurely unenrolled in an experiment, we should add historic experiments to the experiments
object, as well as a flag to each experiment object indicating whether the client is currently enrolled. We should age out experiments after some time, for which my opening bid is one year after a client unenrolls from an experiment. The modified experiment block would look something like:
"experiments": {
"my-cool-experiment": {
"branch": "control",
"type": "normandy-pref-flip",
"isActive": true
},
"my-other-experiment": {
"branch": "treatment",
"type": "normandy-pref-flip",
"isActive": false
}
}
Comment 1•5 years ago
|
||
This will require a bit of design and implementation work:
- it requires persistence of the data first
- it will require some design & implementation for where to put it into outgoing pings
Comment 2•5 years ago
|
||
Sunah, do you know if there is an associated project or required timeline for this?
Updated•5 years ago
|
The project is experiment analysis, which is primarily driven by data science (and this could be considered part of the mozanalysis effort.)
Re: it will require some design & implementation for where to put it into outgoing pings
– this would be a modification to the existing environment.experiments
section, if possible
ni? mythmon has some ideas for making this easier by using normandy's storage mechanism
Comment 5•5 years ago
|
||
Normandy already has a list of all past and current experiments a user has been involved with. When marking experiments as active, it will be easy to mark experiments in a certain state as previously enrolled in the experiment for the duration of the experiment. This will require expanding Normandy's execution model to understand the period of time after unenrollment but before the general experiment ending. This will be a significant change to Normandy.
What Normandy needs from Telemetry is a clear API for how to report this data. It would need to be something that lets us distinguish between
"enrolled and receiving treatment" (what we are currently doing) and "enrolled but no longer receiving treatment" (the new case). Perhaps TelemetryEnvironment.setExperimentActive(experimentName, { type: ..., treatment: false })
?
Either way, if there is an API in Telemetry that we can use to report this data, Normandy can handle generating the data and setting it at startup, just like it does for the existing calls to setExperimentActive
.
Comment 6•5 years ago
|
||
When do you want/need it by? Who do you see owning the design?
I think getting this in for 70 would be nice, since AIUI mythmon is targeting a bunch of fairly significant Normandy changes for that version.
Re: design, I'm fine taking point but it'll be in concert with Felix, Tim and mythmon as stakeholders.
Comment 8•5 years ago
|
||
Alrighty, let me meta this up and file a design bug and assign it to you :)
(( Please do include me in the design as well. Though it might mean putting up with my opinions :S ))
Updated•2 years ago
|
Description
•