1690052 - Potential race condition between Normandy timer and Normandy initialization

Status: RESOLVED FIXED

(Firefox :: Normandy Client, defect, P1)

Description

[Michael Cooper [:mythmon]]

We have been having problems that seem like recipes are running before Normandy proper has initialized. Specifically, during Normandy initialization, the values of default prefs are updated to match their rollout and experiment values. We are getting unenrollments from recipes that seem to imply this happens, and it happens more on computers with fewer CPU cores and slower drives.

The Normandy RecipeRunner runs in response to 3 things:

1. When it is first initialized in some cases, which is strictly after Normandy initializes
2. After RemoteSettings syncs (delayed up to 60 minutes), which is not strictly ordered to be after Normandy initialization
3. When Firefox's update timer procs and calls the recipe runner.

Third third is the one that I expect is getting us most often, since the first is only called synchronously after prefs are applied, and it should be too uncommon for the random delay on the second to be fast enough to cause this problem in the volume we see.

I expect this will fix bug 1663971, but I'm making a separate bug since we aren't sure of the cause of that bug yet, and this may not fix it.

Comment 1

[Michael Cooper [:mythmon]]

Attachment: Bug 1690052 - Wait until Normandy has updated default branch prefs before processing recipes r=gijs

This avoids a potential race condition where the RecipeRunner is triggered before Normandy has fully initialized. When this happens, recipes can wrongly unenroll or graduate, causing many problems and confusing telemetry.

The 30 second delay and 3x max retries were chosen arbitrarily.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b8b16b23dad8
Wait until Normandy has updated default branch prefs before processing recipes r=Gijs

Comment 3

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/b8b16b23dad8