1447324 - Pause Enrollment - Opt-out Studies are not paused

Status: RESOLVED WORKSFORME

(Firefox :: Normandy Client, defect)

Comment 1

[Adrian Florinescu [:aflorinescu]]

[Environments:]
Ubuntu 16.04, Windows 10, 7 , MacOsx10.12

[Affected Versions:]
60.0b4 20180315232954
61.0a1 20180319220116
Not reproducible on 59.0b11 20180219114835

[Description:]
For opt-out Studies the pause enrollment doesn't produce the expected result: pause the enrollment.

[PreRequisites:]
1. Set the app.normandy.dev_mode preference to true to run recipes immediately on startup.
2. Set the app.normandy.logging.level preference to 0 to enable more logging.
3. Set the security.content.signature.root_hash preference to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90.
4. Set the preference value for app.normandy.api_url set to https://normandy.stage.mozaws.net/api/v1


[Steps:]
1. Open Control Center and create an opt-out study recipe type, approve it and publish it.
2. Open FF with pre-requisites, study is installed, close FF.
3. Edit the recipe and turn "Prevent New Enrollment" to true.
4. Use a new profile, setting pre-requisites and open FF.

[Actual Result:]
On 4. the recipe is fetched, executed and the study add-on installed.

[Expected Result:]
On 4. the recipe should be fetched but on execution, it should throw that the enrollment is paused, hence the study add-on should not be installed.

On Shield System add-on (FF59.0b11) the result for this case would be:
1521555891398	extensions.shield-recipe-client.recipe-runner	INFO	Executing recipe "Adi study - add-on blocker" (action=opt-out-study)
1521555891402	extensions.shield-recipe-client.normandy-driver.actions	DEBUG	Enrollment is paused for recipe 206

[Note:]
Heartbeats are missing the "Prevent New Enrollment" option.
"Prevent New Enrollment" option works as expected for the recipe type Pref-Experiments.

Comment 2

[Adrian Florinescu [:aflorinescu]]

Not entirely sure what happened above, but we certainly caught a time window in which the recipe enrollment would be paused in admin, but clients would get the study add-on. On subsequent tries, we could not reproduce this issue for any study recipe or any machine/OS.

Based on the above marking this as WFM for the moment. 

We experienced something similar in the Shield early days, when there was a caching issue that become visible the faster you would edit stuff on admin and then expect it to propagate on the client, but I didn't notice anything of sorts nowadays. 
Mike, any thoughts ?

Comment 3

[Adrian Florinescu [:aflorinescu]]

In case it helps, the time frame corresponds with 2nd revision of recipe 206 from the staging server (https://normandy-admin.stage.mozaws.net/recipe/206/rev/1519bc2558b390111cccf3d0dcbc05089240d6ae79e3e04fecf578844e37bdcb/)

Comment 4

[Michael Cooper [:mythmon]]

> Heartbeats are missing the "Prevent New Enrollment" option.

This makes sense. For preference studies and add-on studies, we need a way to keep users in the study without adding new users. That's what pause enrollment is for. Heartbeat studies don't have users that are in the study, as it is a one time event that happens. As such, there wouldn't be any behavior difference between a disabled and a paused heartbeat recipe. So that option doesn't exist for heartbeat.

> We experienced something similar in the Shield early days, when there was a caching issue that become visible the faster you would edit stuff on admin and then expect it to propagate on the client, but I didn't notice anything of sorts nowadays. 
Mike, any thoughts ?

I think it is probably safe to blame caching here. It sounds like you got a previous version of the recipe for a moment while the caches cleared. It is hard to think this could last long enough to verify on 3 OSs, but It's possible.

I'm going to try and find some data in the wild that shows whether this mechanism is working or not, but it will take some time. I'll update this bug when I find out.