1365680 - Pref Study SHIELD Lazy Client Classification

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Osmose [:osmose, :mkelly]]

The SHIELD system add-on makes a request to the Normandy service to "classify" clients; we geolocate them by IP and return the current time on the server to use as a reliable timestamp. However, many of our recipes do not use this information in their filters.

The extensions.shield-recipe-client.experiments.lazy_classify preference controls whether we make fetching this data lazy or not; if true, then clients do not make the request unless a recipe filter uses values from it. This has the potential to save use a ton of money on server costs, but has the side effect of introducing huge jumps in our server traffic when a new recipe is enabled that uses these values.

We'd like to run a preference experiment to test how our servers can handle the load. The experiment will also help us test preference experiments themselves.

Population Affected
-------------------
Initially, we'd like to run on 100% of Nightly users. Once the code for preference experiments hits Beta (and eventually release), we'd like to expand to those populations. We need a full sample to properly test the effect on server load. 

In the release channel, we will most likely stage rollout to 10%, then 50%, then 100% of the population over a few days.

Effect / Branches
-----------------
The preference "extensions.shield-recipe-client.experiments.lazy_classify" will be set to true for all participants. This will cause the SHIELD add-on to only make the classification request when necessary, instead of the current behavior of always making the request. There are no known user-facing changes.

Cleanup
-------
Once we have confirmed the effect on our server traffic, we will reset the preference to its previous value. If all goes well, we will modify the add-on itself to default the preference to be true.

Goals
-----
1. Measure server traffic to the classify_client URL, filtered by user-agent so we only measure users in the affected channels, and confirm that, at 100% sample with no active recipes using the classify filters, server traffic drops to 5% or lower of our average traffic to the endpoint.

2. Once we hit the release channel, confirm with Services Ops (relud) that our infrastructure can handle the changes in traffic when recipes using the classify filters are enabled and disabled.

Comment 1

[Osmose [:osmose, :mkelly]]

Correction: We're going to drop running this experiment on beta and release, and only run on Nightly. We're also going to drop goal 2 and focus on testing preference experiments themselves and whether we see real-world effects in line with our expectations.

Comment 2

[Osmose [:osmose, :mkelly]]

NEEDINFO Matt_G and rweiss for approval before I am judged by release-drivers. :D

Comment 3

[Matt Grimes [:Matt_G]]

Looks good to me. Signing off!

Comment 4

[[e-mail]]

r+

Comment 5

[Osmose [:osmose, :mkelly]]

NI jcristau to confirm relman approval of this study via release-drivers.

Comment 6

[Julien Cristau [:jcristau]]

yep, all fine with me.

Comment 7

[Osmose [:osmose, :mkelly]]

I think we've got enough useful data from this method of testing. Here's a sheet with the number of requests to the classify endpoint from Nightly users per day, annotated with info about what was affecting the numbers: https://docs.google.com/spreadsheets/d/15pI4yJYdc-qWrYQJsorYdEVFR9uHaIpFcekh-a8ADhM/edit?usp=sharing

The numbers match with the behavior we expect: When the recipe with a country filter was enabled, the request count jumped back up. When the cert we use to signed recipes expired, users stopped running recipes and stopped hitting the endpoint again, but our enrollment didn't increase because new users were not getting the recipe to enroll in the first place. And once the cert fix and cache bust went up, our numbers jumped back up to expected amounts.

The only interesting data that we're lacking is the uptake of the experiment, which I think would be better measured via the telemetry dashboards we have for preference experiments.

Gregglind: Can we disable this experiment on Normandy? Also, how do we run/get access to the automatic study reporting so we can inspect uptake?

Comment 8

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

1.  Disable as you see fit.
2.  Automatic dashboarding is still in-progress.  it has some warts still that need resolve.

Comment 9

[Osmose [:osmose, :mkelly]]

I've disabled the recipe. Looks like we're done here. Thanks!