Closed Bug 1063865 Opened 11 years ago Closed 11 years ago

Determine Sync 1.5 Node Migration Impact

Categories

(Cloud Services :: Operations: Metrics/Monitoring, task)

Product:
Cloud Services
Component:
Operations: Metrics/Monitoring
Platform:
x86
All
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: bobm, Assigned: bobm)

References

Details

(Whiteboard: [qa+])

In the event of a full Sync 1.5 node migration determine the following: 1. The average amount of time it takes for a node migration to complete. 2. The average change in migrations per second over time as it completes.
We should be able to generate this information from processing the 401inator logs.
Whiteboard: [qa+]
The information below was gathered from the 401inator logs for the migrations that took place between September 25th and September 29th. They are of particular interest because they were migrated as whole hosts, instead of by a percentage of users. Durations are here considered to be complete when the traffic first dips below 1QPS for a given hosts averaged over the period of a minute. That is also why the durations are multiples of 60 seconds. sync-74, Users: 54986, Start: Thu Sep 25 19:43:00, Duration: 1620s. sync-79, Users: 54986, Start: Thu Sep 25 19:48:00, Duration: 3600s. sync-82, Users: 54986, Start: Thu Sep 25 19:52:00, Duration: 3480s. sync-84, Users: 54986, Start: Thu Sep 25 20:08:00, Duration: 3540s. sync-90, Users: 54986, Start: Thu Sep 25 21:21:00, Duration: 660s. sync-99, Users: 54986, Start: Thu Sep 25 22:24:00, Duration: 3540s. sync-111, Users: 91272, Start: Thu Sep 25 22:27:00, Duration: 3720s. sync-112, Users: 91272, Start: Thu Sep 25 22:45:00, Duration: 3840s. sync-67, Users: 54986, Start: Fri Sep 26 10:08:00, Duration: 3600s. sync-75, Users: 54986, Start: Fri Sep 26 10:10:00, Duration: 3600s. sync-81, Users: 54986, Start: Fri Sep 26 16:12:00, Duration: 3600s. sync-88, Users: 54986, Start: Fri Sep 26 16:14:00, Duration: 3600s. sync-91, Users: 54986, Start: Fri Sep 26 16:50:00, Duration: 1560s. sync-108, Users: 91272, Start: Fri Sep 26 19:26:00, Duration: 3600s. sync-101, Users: 91272, Start: Mon Sep 29 11:06:00, Duration: 4800s sync-98, Users: 54986, Start: Mon Sep 29 16:56:00, Duration: 2820s Conclusion: With our current settings migrations take about an hour to complete. This likely has to do with a few factors, notably token expiration and our front end supplied Retry-After header are both set to one hour.
Blocks: 1142276
Closing this one for now. Let's re-open if additional metrics related to node migrations are needed.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.