Closed Bug 1246039 Opened 8 years ago Closed 8 years ago

Load test Sync node using AMI 20160216, code-named Heavy Barrel

Categories

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

task
Not set
normal

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: bobm, Assigned: bobm)

Details

Attachments

(1 file)

Build and load test a Sync node running the latest SL6 AMI: 20160119/20, codenamed Gradius.
Node built, and initial first run load test resulted in a 502 spike, which resulted from overflowing the DB connection pools.  See the following log entries:
{"time": "2016-02-04T22:23:39.510962Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a4b9d0> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18756, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T22:23:46.042700Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a45110> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18742, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T22:23:50.716828Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a48110> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18751, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T23:31:37.009508Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a4af90> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 7199, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T23:31:37.048863Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3920290> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 20058, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T23:31:37.056623Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a59f90> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18756, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T23:31:37.063498Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a5ee10> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18751, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T23:31:37.065820Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a44410> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18728, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T23:31:37.073050Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a70150> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18742, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}

This is not a show-stopper as is, but requires comparison to other nodes under similar load.
Not much to see here, but  Bob observed a spike in 5xx responses in the midst of this test, and will investigate.  Timestamp in the log is in UTC.
Changing this ticket to load test latest SL6 AMI: Heavy Barrel
Summary: Load test Sync node using AMI SL6 20160119/20, codenamed Gradius → Load test Sync node using AMI 20160216, code-named Heavy Barrel
Load test completed successfully.
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Verified.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: