1443940 - [Shield] Pref Flip Study: Normandy timer speed up

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Michael Cooper [:mythmon]]

Basic description of experiment:
Currently Normandy checks for new recipes once every 24 hours. We'd like to change this to once every 6 hours. This will give us faster enrollment and unenrollment from studies.
Why 6 hours? It's mostly arbitrary. My reasoning is that power-of-two steps are nice, and that 6 hours starts to put Normandy into the "answers in one working day" range.

What is the preference we will be changing?
On Fx60 and above: app.normandy.run_interval_seconds
On Fx59 and below: extensions.shield-recipe-client.run_interval_seconds

What are the branches of the study and what values should each branch be set to?
Experiment
Control

What percentage of users do you want in each branch?
TBD

What Channels and locales do you intend to ship to?
There are no channel/locale concerns

What is your intended go live date and how long will the study run?
Start as soon as possible
End no later than when Nightly becomes Firefox 62 (May 7th)
End likely sooner, once we have enough data about the server load

Are there specific criteria for participants?
No

What is the main effect you are looking for and what data will you use to make these decisions? 
Datadog and other server statistics
Telemetry events for Normandy enrollment/unenrollment

Who is the owner of the data analysis for this study?
TBD

Will this experiment require uplift?
No

QA Status of your code:
TBD

Do you plan on surveying users at the end of the study? 
No

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:
Bug 1443911

Comment 1

[Matt Grimes [:Matt_G]]

This all looks straight forward to me. If you want to start in Nightly you can send the intent to ship and we can get it out this coming Monday.

Comment 2

[Robert (rrayborn, he/him)]

We're live with 50% of Nightly.

Comment 3

[Robert (rrayborn, he/him)]

Mythmon - when do you want this killed, it's been a week right now.

Comment 4

[Michael Cooper [:mythmon]]

I was hoping to be able to ship the beta version of the study this week, so they could run concurrently, but that didn't happen.

I think the data we wanted here is available now, in the server logs, and in telemetry. I think we can shut this down now until we're ready for beta, and I'll work on seeing if the data agrees with what we expected.

As a side note, hopefully unenrollment will be very fast here, since all the clients that need to unenroll are on 6 hour times. We should be able to see this in the unenrollment curve in telemetry events.

Comment 5

[Matt Grimes [:Matt_G]]

Done. The study has ended on Nightly.

Comment 6

[Michael Cooper [:mythmon]]

We'd like to run this study on Beta now. We've gone through formal QA by Adrian and gotten sign off from him and from Relman (via jcristau) in email.

Julien, Adrian, can you copy your sign off to this bug, for completeness sake?

Rhelmer: Studies on beta also require Firefox peer sign-off for the code as well. You reviewed this code in https://github.com/mozilla/normandy/pull/641. Can you give your sign-off as a Firefox peer to run this pref flip study?

Comment 7

[Robert Helmer [:rhelmer]]

(In reply to Michael Cooper [:mythmon] from comment #6)
> We'd like to run this study on Beta now. We've gone through formal QA by
> Adrian and gotten sign off from him and from Relman (via jcristau) in email.
> 
> Julien, Adrian, can you copy your sign off to this bug, for completeness
> sake?
> 
> Rhelmer: Studies on beta also require Firefox peer sign-off for the code as
> well. You reviewed this code in
> https://github.com/mozilla/normandy/pull/641. Can you give your sign-off as
> a Firefox peer to run this pref flip study?

r+

Comment 8

[Adrian Florinescu [:aflorinescu]]

(In reply to Michael Cooper [:mythmon] from comment #6)
> We'd like to run this study on Beta now. We've gone through formal QA by
> Adrian and gotten sign off from him and from Relman (via jcristau) in email.
> 
> Julien, Adrian, can you copy your sign off to this bug, for completeness
> sake?
> 
> Rhelmer: Studies on beta also require Firefox peer sign-off for the code as
> well. You reviewed this code in
> https://github.com/mozilla/normandy/pull/641. Can you give your sign-off as
> a Firefox peer to run this pref flip study?

r+

Comment 9

[Julien Cristau [:jcristau]]

ship it!

Comment 10

[Robert (rrayborn, he/him)]

We're live, starting at 10% of Beta, targeting 50% by EOW.

Comment 11

[Adrian Florinescu [:aflorinescu]]

Any updates here?

Comment 12

[Michael Cooper [:mythmon]]

This got delayed, and just went to 50% of beta today. We'll let it run for a few days before turning it off, to see if the servers notice.

Comment 13

[Michael Cooper [:mythmon]]

This has been running at 50% of beta for a few days now. I think this is the best test that we're going to get for now, and would like to call this done.

Rob, can we turn off this experiment?

Comment 14

[Matt Grimes [:Matt_G]]

Done. Thanks!