1500230 - Test preference study to verify Normandy preference setting

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Michael Cooper [:mythmon]]

In bug 1495152, we learned that in a recent study (bug 1474484), about a quarter of users that Normandy reported were enrolled in the experiment (via the Telemetry's experiments field) did not report the preference as being set (according to Telemetry's pref tracking).

After exhausting our options trying to manually reproduce this or discover the problem from the code, we'd like to try and reproduce it in the wild with a dummy pref.

The preferences in question are "app.normandy.test.with_default" and "app.normandy.test.without_default", which are both boolean prefs. "with_default" has a default value of false. We'll make two recipes that change the value of these two preferences to true on the user branch. Both of these preferences are reported via Telemetry, so if the problem is related to either Normandy or Telemetry, we should be able to reproduce the problem seen in the WebRender study.

# Basic description of experiment
Set two dummy prefs and observe their values in Telemetry.

# What are the branches of the study?
For each pref, we want an independent branch that sets the value to either true or false, for a total of four branches (across two recipes).

# What percentage of users do you want in each branch?
Evenly split.

# What Channels and locales do you intend to ship to?
Nightly only.

# What is your intended go live date and how long will the study run?
ASAP. One week to start, though we may extend the window if we need further information.

# Are there specific criteria for participants?
Build IDs greater-than-or-equal to 20181018123730.

# What is the main effect you are looking for and what data will you use to make these decisions?
Is the enrollment problem seen in the WebRender study reproduceable? We'll be looking at Normandy enrollment and unenrollment events, the value of the experiments field in the environment of the main ping, and the value of the dummy preferences reported by Telemetry

# Who is the owner of the data analysis for this study? David Bolter, Tim Smith

# Who will have access to the data?
Chris Pearce, Michael Cooper, 

# Do you plan on surveying users at the end of the study?
No

# User facing title of the experiment:
"Normandy verification test (bug X)" where X is this bug number.

# User facing description of the experiment: Test that Normandy can correctly update preferences in Firefox.

# Link to any relevant google docs / Drive files that describe the project:
Bug 1499552, bug 1495152

Comment 1

[Michael Cooper [:mythmon]]

The preferences being set have changed. The preferences are now

- app.normandy.test.with_true_default
- app.normandy.test.with_false_default
- app.normandy.test.without_default

There should be 3 distinct recipes--one for each preference--, each with evenly split branches between true and false.

Comment 2

[Michael Cooper [:mythmon]]

I've created three recipes for this study. I've targeted them at 3 non-overlapping 10% samples of Nightly with the correct buildIds.

Default false: https://delivery-console.prod.mozaws.net/recipe/614/
Default true: https://delivery-console.prod.mozaws.net/recipe/615/
No default: https://delivery-console.prod.mozaws.net/recipe/616/

Matt, can you review these recipes?

Comment 3

[Matt Grimes [:Matt_G]]

Approved and published.

Comment 4

[Michael Cooper [:mythmon]]

Collecting data over the weekend showed all three of these recipes behaving as expected.

Chris Pearce ran analysis of the recipes, here: https://gist.github.com/cpearce/6c916fb4825c868f7a6d3acb5b7662ef

To summarize, we saw less than 0.1% of users enrolled in the study that did not have the expected telemetry set for the user preferences reporting. This shows that the problem we saw with WebRender is not reproduced in the dummy pref test.

Given the above, I have disabled the recipes from comment 2, ending the experiments. Users will unenroll as they check back in to Normandy.