Closed
Bug 1443940
Opened 7 years ago
Closed 7 years ago
[Shield] Pref Flip Study: Normandy timer speed up
Categories
(Shield :: Shield Study, enhancement)
Shield
Shield Study
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: mythmon, Assigned: mythmon)
References
Details
Basic description of experiment:
Currently Normandy checks for new recipes once every 24 hours. We'd like to change this to once every 6 hours. This will give us faster enrollment and unenrollment from studies.
Why 6 hours? It's mostly arbitrary. My reasoning is that power-of-two steps are nice, and that 6 hours starts to put Normandy into the "answers in one working day" range.
What is the preference we will be changing?
On Fx60 and above: app.normandy.run_interval_seconds
On Fx59 and below: extensions.shield-recipe-client.run_interval_seconds
What are the branches of the study and what values should each branch be set to?
Experiment
Control
What percentage of users do you want in each branch?
TBD
What Channels and locales do you intend to ship to?
There are no channel/locale concerns
What is your intended go live date and how long will the study run?
Start as soon as possible
End no later than when Nightly becomes Firefox 62 (May 7th)
End likely sooner, once we have enough data about the server load
Are there specific criteria for participants?
No
What is the main effect you are looking for and what data will you use to make these decisions?
Datadog and other server statistics
Telemetry events for Normandy enrollment/unenrollment
Who is the owner of the data analysis for this study?
TBD
Will this experiment require uplift?
No
QA Status of your code:
TBD
Do you plan on surveying users at the end of the study?
No
Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:
Bug 1443911
Flags: needinfo?(mgrimes)
Updated•7 years ago
|
Flags: needinfo?(mgrimes)
Comment 1•7 years ago
|
||
This all looks straight forward to me. If you want to start in Nightly you can send the intent to ship and we can get it out this coming Monday.
Comment 2•7 years ago
|
||
We're live with 50% of Nightly.
Comment 3•7 years ago
|
||
Mythmon - when do you want this killed, it's been a week right now.
Flags: needinfo?(mcooper)
Assignee | ||
Comment 4•7 years ago
|
||
I was hoping to be able to ship the beta version of the study this week, so they could run concurrently, but that didn't happen.
I think the data we wanted here is available now, in the server logs, and in telemetry. I think we can shut this down now until we're ready for beta, and I'll work on seeing if the data agrees with what we expected.
As a side note, hopefully unenrollment will be very fast here, since all the clients that need to unenroll are on 6 hour times. We should be able to see this in the unenrollment curve in telemetry events.
Flags: needinfo?(mcooper)
Comment 5•7 years ago
|
||
Done. The study has ended on Nightly.
Assignee | ||
Comment 6•7 years ago
|
||
We'd like to run this study on Beta now. We've gone through formal QA by Adrian and gotten sign off from him and from Relman (via jcristau) in email.
Julien, Adrian, can you copy your sign off to this bug, for completeness sake?
Rhelmer: Studies on beta also require Firefox peer sign-off for the code as well. You reviewed this code in https://github.com/mozilla/normandy/pull/641. Can you give your sign-off as a Firefox peer to run this pref flip study?
Flags: needinfo?(rhelmer)
Flags: needinfo?(jcristau)
Flags: needinfo?(adrian.florinescu)
Comment 7•7 years ago
|
||
(In reply to Michael Cooper [:mythmon] from comment #6)
> We'd like to run this study on Beta now. We've gone through formal QA by
> Adrian and gotten sign off from him and from Relman (via jcristau) in email.
>
> Julien, Adrian, can you copy your sign off to this bug, for completeness
> sake?
>
> Rhelmer: Studies on beta also require Firefox peer sign-off for the code as
> well. You reviewed this code in
> https://github.com/mozilla/normandy/pull/641. Can you give your sign-off as
> a Firefox peer to run this pref flip study?
r+
Flags: needinfo?(rhelmer)
Comment 8•7 years ago
|
||
(In reply to Michael Cooper [:mythmon] from comment #6)
> We'd like to run this study on Beta now. We've gone through formal QA by
> Adrian and gotten sign off from him and from Relman (via jcristau) in email.
>
> Julien, Adrian, can you copy your sign off to this bug, for completeness
> sake?
>
> Rhelmer: Studies on beta also require Firefox peer sign-off for the code as
> well. You reviewed this code in
> https://github.com/mozilla/normandy/pull/641. Can you give your sign-off as
> a Firefox peer to run this pref flip study?
r+
Flags: needinfo?(adrian.florinescu)
Comment 10•7 years ago
|
||
We're live, starting at 10% of Beta, targeting 50% by EOW.
Comment 11•7 years ago
|
||
Any updates here?
Assignee | ||
Comment 12•7 years ago
|
||
This got delayed, and just went to 50% of beta today. We'll let it run for a few days before turning it off, to see if the servers notice.
Assignee | ||
Comment 13•7 years ago
|
||
This has been running at 50% of beta for a few days now. I think this is the best test that we're going to get for now, and would like to call this done.
Rob, can we turn off this experiment?
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Comment 14•7 years ago
|
||
Done. Thanks!
You need to log in
before you can comment on or make changes to this bug.
Description
•