Load test Sync node using AMI 20160216, code-named Heavy Barrel

VERIFIED FIXED

Status

VERIFIED FIXED
3 years ago
3 years ago

People

(Reporter: bobm, Assigned: bobm)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

3 years ago
Build and load test a Sync node running the latest SL6 AMI: 20160119/20, codenamed Gradius.
(Assignee)

Comment 1

3 years ago
Node built, and initial first run load test resulted in a 502 spike, which resulted from overflowing the DB connection pools.  See the following log entries:
{"time": "2016-02-04T22:23:39.510962Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a4b9d0> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18756, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T22:23:46.042700Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a45110> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18742, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T22:23:50.716828Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a48110> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18751, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T23:31:37.009508Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a4af90> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 7199, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T23:31:37.048863Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3920290> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 20058, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T23:31:37.056623Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a59f90> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18756, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T23:31:37.063498Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a5ee10> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18751, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T23:31:37.065820Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a44410> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18728, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}
{"time": "2016-02-04T23:31:37.073050Z", "v": 1, "message": "Connection <pymysql.connections.Connection object at 0x3a70150> exceeded timeout; recycling", "hostname": "ip-172-31-17-164", "pid": 18742, "op": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog", "name": "syncstorage.storage.sql.dbconnect.QueuePoolWithMaxBacklog"}

This is not a show-stopper as is, but requires comparison to other nodes under similar load.
Created attachment 8716420 [details]
Load test logs for new sync node, 2016-02-04

Not much to see here, but  Bob observed a spike in 5xx responses in the midst of this test, and will investigate.  Timestamp in the log is in UTC.
(Assignee)

Comment 3

3 years ago
Changing this ticket to load test latest SL6 AMI: Heavy Barrel
Summary: Load test Sync node using AMI SL6 20160119/20, codenamed Gradius → Load test Sync node using AMI 20160216, code-named Heavy Barrel
(Assignee)

Comment 4

3 years ago
Load test completed successfully.
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Verified.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.