[Experiment] Staged Rollout: Increasing Normandy skew out changes from 15 minutes to 60 minutes Fx 73.0 to 75.0 Release
Categories
(Shield :: Shield Study, task, P3)
Tracking
(Not tracked)
People
(Reporter: experimenter, Assigned: shell)
References
()
Details
Increasing Normandy skew out changes from 15 minutes to 60 minutes
Normandy is going to be supporting add-on rollouts and hotfixes. When adding new actions that go out to a large population % at once - we want to limit potential risk of overloading our telemetry pipeline or cloud load balancing services.
--Normandy changes go out through push.
--The Normandy client skews changes to apply over 15 minutes. We'd like to increase that to 60 minutes (more in line with how firefox skews out updates).
--When Normandy makes changes to the client- it triggers a main ping (since it is an environment change). It is those pings that we want to stagger to avoid DoS ourselves with large % changes.
Rollouts typically go to 25%-50-100 or 25%-75-100, but for some situations (marketing campaigns or future hotfixes) going to 100% right away is a valid scenario. We looked at recent pref rollouts to see if those changes impacted our back end and found that we had seen heavy spikes (enough to page Wesley and a throw a few 500 errors). While the failures were graceful - we want to avoid that.
This change would also help potential secondary systems that we might hit (ex: locale look-up or an backend services the features in the rollout use).
Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.
More information: https://experimenter.services.mozilla.com/experiments/increasing-normandy-spreading-out-changes-from-15-minutes-to-60-minutes/
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Comment 1•5 years ago
|
||
|
|
Reporter | |
Updated•5 years ago
|
Description
•