Closed Bug 1633007 Opened 2 years ago Closed 9 months ago

Consider two-phase evaluation on targeting for first-run experiment enrollments

Categories

(Firefox :: Nimbus Desktop Client, enhancement, P3)

enhancement

Tracking

()

RESOLVED WONTFIX
Tracking Status
firefox77 --- wontfix
firefox78 --- fix-optional
firefox79 --- fix-optional

People

(Reporter: nanj, Unassigned)

References

(Blocks 1 open bug)

Details

In various first-run related experiments, we have observed a race-condition between the experiment enrollment and the browser startup, which has caused several anomalies and biased the experiment results.

While we are still trying to understand the root cause of the race condition, we could use some preventions to mitigate the race condition during the experiment enrollment. In particular, the two-phase evaluation.

The key observation is that when enrolling a user into an experiment, its targeting criteria (invariant) should hold true in the whole process of the enrollment. In the current implementation, we only check that once at the beginning of the enrollment. However, the whole enrollment takes several steps to finish, and the invariant could be violated during that time window. We could ensure that by checking the invariant again right before we register the enrollment. Upon violations, we simply record the failure and abort the enrollment.

For example, for a given recipe, we could do

if (isMatch(recipe.targetingString)) {
  // begin an enrollment 

  // do some internal bookkeeping
   ...

  // checking the targeting again before submitting to Normandy
  if (isMatch(recipe.targetingString)) {
     normandy.setActiveExperiment(recipe) 
  } else {
     // abort the enrollment, undo the internal changes
  }
}

Note:

  • This assumes the recipe targeting could be used as the invariant, which is usually the case (e.g. trailhead.firstrun.didSeeAboutWelcome|preferenceValue for first exps)
  • The "undo" part could be tricky to implement depending on how many internal states need to be restored
  • Another solution to this race condition is - "lazy" enrollment, i.e. we defer the enrollment until we're sure the expected effect has been applied for that experiment. For instance, the user has seen an "about:welcome" variant with the confirmation of such impression. This is the most reliable approach, but also not trivial to implement
Priority: -- → P2
Priority: P2 → --
Priority: P2 → --
Priority: -- → P2
Blocks: 1641286
Priority: P2 → P3
Component: Messaging System → Nimbus Desktop Client

I don't think this is still an issue for us given that we have an alternative way to target first run users, please reopen if we want this still

Status: NEW → RESOLVED
Closed: 9 months ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.