Open Bug 1549760 Opened 3 years ago Updated 3 years ago

[meta] Send historical experiments information over Telemetry

Categories

(Toolkit :: Telemetry, enhancement, P3)

enhancement

Tracking

()

People

(Reporter: bugzilla, Unassigned)

References

Details

(Keywords: meta)

This has been discussed a bit in the past as a desirable thing but in particular for ease of analysis of clients that are prematurely unenrolled in an experiment, we should add historic experiments to the experiments object, as well as a flag to each experiment object indicating whether the client is currently enrolled. We should age out experiments after some time, for which my opening bid is one year after a client unenrolls from an experiment. The modified experiment block would look something like:

"experiments": {
  "my-cool-experiment": {
    "branch": "control",
    "type": "normandy-pref-flip",
    "isActive": true
  },
  "my-other-experiment": {
    "branch": "treatment",
    "type": "normandy-pref-flip",
    "isActive": false
  }
}

This will require a bit of design and implementation work:

  • it requires persistence of the data first
  • it will require some design & implementation for where to put it into outgoing pings

Sunah, do you know if there is an associated project or required timeline for this?

Flags: needinfo?(ssuh)

The project is experiment analysis, which is primarily driven by data science (and this could be considered part of the mozanalysis effort.)

Re: it will require some design & implementation for where to put it into outgoing pings – this would be a modification to the existing environment.experiments section, if possible

Flags: needinfo?(ssuh)

ni? mythmon has some ideas for making this easier by using normandy's storage mechanism

Flags: needinfo?(mcooper)

Normandy already has a list of all past and current experiments a user has been involved with. When marking experiments as active, it will be easy to mark experiments in a certain state as previously enrolled in the experiment for the duration of the experiment. This will require expanding Normandy's execution model to understand the period of time after unenrollment but before the general experiment ending. This will be a significant change to Normandy.

What Normandy needs from Telemetry is a clear API for how to report this data. It would need to be something that lets us distinguish between
"enrolled and receiving treatment" (what we are currently doing) and "enrolled but no longer receiving treatment" (the new case). Perhaps TelemetryEnvironment.setExperimentActive(experimentName, { type: ..., treatment: false })?

Either way, if there is an API in Telemetry that we can use to report this data, Normandy can handle generating the data and setting it at startup, just like it does for the existing calls to setExperimentActive.

Flags: needinfo?(mcooper)

When do you want/need it by? Who do you see owning the design?

Flags: needinfo?(ssuh)

I think getting this in for 70 would be nice, since AIUI mythmon is targeting a bunch of fairly significant Normandy changes for that version.

Re: design, I'm fine taking point but it'll be in concert with Felix, Tim and mythmon as stakeholders.

Flags: needinfo?(ssuh)

Alrighty, let me meta this up and file a design bug and assign it to you :)

(( Please do include me in the design as well. Though it might mean putting up with my opinions :S ))

Keywords: meta
Priority: -- → P3
Summary: Add historical experiments to `experiments` object in environment → [meta] Send historical experiments information over Telemetry
Depends on: 1557757
You need to log in before you can comment on or make changes to this bug.