1506221 - Don't allow enrollment in default-branch pref experiments when the pref doesn't have a built-in value

Status: NEW

(Firefox :: Normandy Client, defect, P3)

Description

[Michael Cooper [:mythmon]]

In bug 1502410, we discovered that, among other things, unenrolling from a preference experiment that didn't previously have a value doesn't work in various ways. Before 1502410, the experiment simply never unenrolled (a regression from bug 1471025). After the fix in 1502410 lands, the unenrollment requires a Firefox restart, which can leave Firefox in a weird state where Normandy believes it has unenrolled the user, but the pref is still set to the experimental value.

Although this type of study has been allowed in the past, it seems that the prudent thing to do now, at least while the above is true, is to disallow experiments on these preferences. Depending on the outcome of 1505941, we may also revert this change.

To do this, I propose that

* If a preference experiments would enroll a user in a preference experiment that sets a preference for which there is no built-in value
* then Normandy should
  * Not enroll the user.
  * Not store any state about the experiment (in case the pref exists in a later version).
  * Send an `enrollFailed` telemetry event.

We probably don't want to do this immediately, as this is a policy change on the type of experiments we can run. I expect we'll want to target this late-ish in the Firefox 66 cycle, or in Firefox 67.

Comment 1

[Michael Cooper [:mythmon]]

In a recent study (bug 1528310) we learned that in the case that a pref does not have a default branch value but has a user branch value, enrollment fails entirely without sending an error. This is worse than originally reported.