1634842 - Sampling capability for maintaining a long term no-show hold out set

Status: RESOLVED WONTFIX

(Firefox :: Nimbus Desktop Client, enhancement, P5)

Description

[Su-Young Hong]

Summary: (updated for clarity 7/23)

We want the ability the run long-term hold out studies (specifically for no-message experiments).

We might be able to implement this by just running a normal "no-message" holdout experiment for a long time using the normal experiment deployment.

However, I'd like to make sure this deployment is robust (in general), and won't fail (or mitigate chance of failure) in the following cases:

• human error
  • we delete the recipe from the server by accident
  • we accidentally re-use the same "bucket space" that we used in the still-running holdout experiment
  • we run through the entire "bucket space", and then reseed a new bucket space (meaning some users in each bucket will be in the still-running hold out experiment
• server error
  • server goes down for a bit (this has happened before) and users don't get their recipe update

Maybe the current messaging experiment system is robust against this, but if there's anything we can do to mitigate, we should

========================================== original bug below ==========================================

When we release the fast experiments platform to stakeholders, we will want to launch a long - term hold out study for no messaging.

Study requirements:

• hold out set (X% of users over time - open ended enrollment period)
• this hold out set gets no messages for any of the messaging layers / groups
• users in this set don't get enrolled into any messaging experiments

We'll need to add some capability to accomplish this, specifically, be able to enroll and keep users into this hold out set, while simultaneously excluding them from all future experiments, while executing normal messaging experiments without worrying about this hold out set.

It might be a good idea to implement this independent of remote settings / experiment manager, and just hard-code this experiment into the code to guarantee the above.

Comment 1

[Kate Hudson :k88hudson]

Is this required for any of the shirely experiments?

Comment 2

[:shell escalante]

Jim is working with Shirley on normal CFR experiments. For Shirley this will not be used - but should be added to future requirements.

Comment 3

[Barret Rennie [:barret] (they/them)]

This is being implemented as part of https://mozilla-hub.atlassian.net/browse/EXP-3946