699294 - nagios reports intermittent "too many connections" SQL failures

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(not sure if this is correct component, please reassign as needed.)


At 13:17 PDT today, we hit a bunch of nagios alerts today about "too many sql connections" impacting most? all? buildbot-masters. These all cleared by 13:24. 
From a quick scan of my email, these also happened on the 26th, 18th Oct, but there may have been other occasions.

Whats going on here? Is this a genuine problem with the db server, or should the nagios settings be tweaked?


Example:
>> Subject: ** PROBLEM alert - buildbot-master3.build.mtv1/MySQL
>> connectivity is CRITICAL **
>> Date: Wed,  2 Nov 2011 13:17:51 -0700 (PDT)
>> From: [e-mail] (nagios)
>> To: [e-mail],[e-mail]
>>
>> ***** Nagios  *****
>>
>> Notification Type: PROBLEM
>>
>> Service: MySQL connectivity
>> Host: buildbot-master3.build.mtv1
>> Address: 10.250.48.236
>> State: CRITICAL
>>
>> Date/Time: 11-02-2011 13:17:51
>>
>> Additional Info:
>>
>> Too many connections
>>
>>
>

Comment 1

[Amy Rich [:arr] [:arich]]

buildduty and I discussed this in irc while it was happening.  The database server ran out of memory and started swapping.  Killing off a database dump cleared things up.