Closed
Bug 1246039
Opened 8 years ago
Closed 8 years ago
Load test Sync node using AMI 20160216, code-named Heavy Barrel
Categories
(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)
Cloud Services
Operations: Deployment Requests - DEPRECATED
Tracking
(Not tracked)
VERIFIED
FIXED
People
(Reporter: bobm, Assigned: bobm)
Details
Attachments
(1 file)
693 bytes,
text/plain
|
Details |
Build and load test a Sync node running the latest SL6 AMI: 20160119/20, codenamed Gradius.
Assignee | ||
Comment 1•8 years ago
|
||
Node built, and initial first run load test resulted in a 502 spike, which resulted from overflowing the DB connection pools. See the following log entries: {"time": "2016-02-04T22:23:39.510962Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a4b9d0> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18756, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"} {"time": "2016-02-04T22:23:46.042700Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a45110> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18742, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"} {"time": "2016-02-04T22:23:50.716828Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a48110> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18751, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"} {"time": "2016-02-04T23:31:37.009508Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a4af90> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 7199, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"} {"time": "2016-02-04T23:31:37.048863Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3920290> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 20058, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"} {"time": "2016-02-04T23:31:37.056623Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a59f90> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18756, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"} {"time": "2016-02-04T23:31:37.063498Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a5ee10> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18751, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"} {"time": "2016-02-04T23:31:37.065820Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a44410> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18728, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"} {"time": "2016-02-04T23:31:37.073050Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a70150> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18742, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"} This is not a show-stopper as is, but requires comparison to other nodes under similar load.
Comment 2•8 years ago
|
||
Not much to see here, but Bob observed a spike in 5xx responses in the midst of this test, and will investigate. Timestamp in the log is in UTC.
Assignee | ||
Comment 3•8 years ago
|
||
Changing this ticket to load test latest SL6 AMI: Heavy Barrel
Summary: Load test Sync node using AMI SL6 20160119/20, codenamed Gradius → Load test Sync node using AMI 20160216, code-named Heavy Barrel
Assignee | ||
Comment 4•8 years ago
|
||
Load test completed successfully.
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•