1394962 - [Shield] Opt-out Study Nightly 57: Enrollment Test

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Matt Grimes [:Matt_G]]

This is to test the enrollment function of opt-out studies. We'll be deploying a study with no UI surface and no branches. It will enroll users, write to telemetry that the user has been enrolled, and then expire. 

For this test we'll use:
40% of Nightly 57 in en_US 


We'll be validating population size, locale, and that the telemetry pipeline is working as expected.

Intended ship date: 08/30/2017
End date: 09/01/2017

Comment 1

[Kamyar Ardekani (he/him)]

Enrollment status:

We see 13190 distinct clientIds enrolled based on Telemetry experiment tagging and 12467 install pings coming from the addon

Comment 2

[Matt Grimes [:Matt_G]]

Looks good to me. Closing this bug.

Comment 3

[Osmose [:osmose, :mkelly]]

I'd actually like to keep this open until we can verify that users are unenrolled after the experiment ends. Kamyar is already tracking this.

Comment 4

[Kamyar Ardekani (he/him)]

Based on the pings received from Sept 12 to Sept 20, at most 51 clients still have the addon installed => https://gist.github.com/raymak/d18ed8fab091a7bf92c8a2ce17eb7191

Comment 5

[Osmose [:osmose, :mkelly]]

(In reply to Kamyar Ardekani from comment #4)
> Based on the pings received from Sept 12 to Sept 20, at most 51 clients
> still have the addon installed =>
> https://gist.github.com/raymak/d18ed8fab091a7bf92c8a2ce17eb7191

That could be due to the built-in failsafe triggering on the 12th, though. I'd like to confirm whether the unenrollment mechanism from disabling the recipe worked. Can you get the number of people who had the study installed on the 10th of September?

Comment 6

[Kamyar Ardekani (he/him)]

Looking at pings from the 10th, 16 clients have it. Looking at the 9th and 10th, 27 clients.

Comment 7

[Osmose [:osmose, :mkelly]]

Cool, thanks for checking! I am now convinced that we are able to rollback studies. :D