Open Bug 1634842 Opened 3 years ago Updated 8 months ago

Sampling capability for maintaining a long term no-show hold out set

Categories

(Firefox :: Nimbus Desktop Client, enhancement, P5)

enhancement

Tracking

()

People

(Reporter: shong, Unassigned)

References

(Depends on 1 open bug)

Details

Summary: (updated for clarity 7/23)

We want the ability the run long-term hold out studies (specifically for no-message experiments).

We might be able to implement this by just running a normal "no-message" holdout experiment for a long time using the normal experiment deployment.

However, I'd like to make sure this deployment is robust (in general), and won't fail (or mitigate chance of failure) in the following cases:

  • human error
    • we delete the recipe from the server by accident
    • we accidentally re-use the same "bucket space" that we used in the still-running holdout experiment
    • we run through the entire "bucket space", and then reseed a new bucket space (meaning some users in each bucket will be in the still-running hold out experiment
  • server error
    • server goes down for a bit (this has happened before) and users don't get their recipe update

Maybe the current messaging experiment system is robust against this, but if there's anything we can do to mitigate, we should

========================================== original bug below ==========================================

When we release the fast experiments platform to stakeholders, we will want to launch a long - term hold out study for no messaging.

Study requirements:

  • hold out set (X% of users over time - open ended enrollment period)
  • this hold out set gets no messages for any of the messaging layers / groups
  • users in this set don't get enrolled into any messaging experiments

We'll need to add some capability to accomplish this, specifically, be able to enroll and keep users into this hold out set, while simultaneously excluding them from all future experiments, while executing normal messaging experiments without worrying about this hold out set.

It might be a good idea to implement this independent of remote settings / experiment manager, and just hard-code this experiment into the code to guarantee the above.

Depends on: 1634841
Blocks: x-man
Iteration: --- → 78.1 - May 4 - May 17
Priority: -- → P2
Iteration: 78.1 - May 4 - May 17 → 78.2 - May 18 - May 31
Iteration: 78.2 - May 18 - May 31 → 79.1 - June 1 - June 14
Iteration: 79.1 - June 1 - June 14 → 79.2 - June 15 - June 28
Iteration: 79.2 - June 15 - June 28 → 80.1 - June 29 - July 12
Priority: P2 → --

Is this required for any of the shirely experiments?

Flags: needinfo?(sescalante)
Iteration: 80.1 - June 29 - July 12 → 82.1 - Aug 24 - Sep 6
Priority: -- → P2

Jim is working with Shirley on normal CFR experiments. For Shirley this will not be used - but should be added to future requirements.

Flags: needinfo?(sescalante)
Iteration: 82.1 - Aug 24 - Sep 6 → 83.1 - Sept 21 - Oct 4
Priority: P2 → P3
Component: Messaging System → Nimbus Desktop Client
Priority: P3 → P5
Iteration: 83.1 - Sept 21 - Oct 4 → ---
You need to log in before you can comment on or make changes to this bug.