1634837 - Create Guardrails in Experiment Manager / Messaging Systems Experiment Launch Process

Status: RESOLVED WORKSFORME

(Firefox :: Messaging System, enhancement, P2)

Description

[Su-Young Hong]

Summary

Currently, the experiment manager / messaging systems experiment launch process is

• deployer writes up the experiment recipe
• it gets reviewed by a peer
• merges to server

Since we're relying entirely on human oversight, there's great systematic risk for mis-configured recipes to leak out. For example:

• for a beta experiment, channel filter is not set, and a huge amount of release users see the experiment.

We should be as conservative as possible for preventing these kind of mishaps since, not only will this affect general user experience, but we could also end up affecting the results of other experiments in progress, as well as the viability of future experiments (i.e. we accidentally send a beta CFR experiment to release. We also want to run this experiment in release later. But due to this mishap, a significant amount of release users saw the treatment, affecting validity of release experiment results).

Recommendation

We need some sort of guardrail system to minimize the risk of misconfigured, or insufficiently configured recipes to hit production.

• require certain fields the messaging system recipes to be filled in order for recipe to launch
  • release channel
  • version[*]
• have some convention that gives us fairly strong confidence that the experiment slug is unique (currently, experiment slug is set arbitrarily by the deployer).

I think these come "free" from the experimenter process that's currently in place for normandy. Piggybacking off of that could be a good idea.

[*] note, I think having an implicit "greater than" default convention for versions would be a good idea, since then, by default, we don't have to worry about experiments crossing release cycles.

Comment 1

[Hanna Alemu [:hanna_a]]

@Punam is this bug still relevant? If so, it seems like it should be forwarded to the experimenter team instead? wdyt?

Comment 2

[Punam Dahiya [:pdahiya]]

With Nimbus experimenter as main source of deploying experiment, ask in the original description are addressed and bug should be safe to close. Please feel free to reopen if there is additional work scoped in this enhancement.