1006031 - SQL Connection Pool Monitoring for Sync 1.5

Status: RESOLVED FIXED

(Cloud Services :: Operations: Metrics/Monitoring, task)

Description

[Bob Micheletto [:bobm]]

Add SQL Connection pool monitoring for Sync 1.5 production similar to what is in place for Sync 1.1.

Comment 1

[Ryan Kelly [:rfkelly]]

Bob, please let me know what you need from the code side here, if anything.

Comment 2

[Bob Micheletto [:bobm]]

(In reply to Ryan Kelly [:rfkelly] from comment #1)
> Bob, please let me know what you need from the code side here, if anything.

Does the Sync 1.5 code output SQL pool events to statsd like the Sync 1.1 code does?  I'm thinking of Connection Pool recycling events, Connection Pool Max Backlog Reached, and Connection Pool Timings.  Perhaps most of these are already in the app log?

Comment 3

[Ryan Kelly [:rfkelly]]

The request summary log line should include a timer "syncstorage.storage.sql.pool.get" giving the total time spent waiting for a connection from the pool.  This will already be going into heka as part of the json logging data.

The recycling and max-backlog events will be output to the json app log as separate messages, in fact I think :jbonacci has noted them in some of his debug work in the past.  But I can probably make them easier to collect, by including them in the summary log in the same manner as the timer.  I filed Bug 1053050 to investigate this.

Does including all this stuff into a single json log line make sense?  Or would you prefer they go out as individual messages as they occur?

Comment 4

[Bob Micheletto [:bobm]]

Removing the infrastructure limitation on this bug.

If all the information is in the app logs now, we can probably build a dashboard for looking at these as is.