If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Buildbot DB max connections

RESOLVED FIXED

Status

Data & BI Services Team
DB: MySQL
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: Callek, Unassigned)

Tracking

Department-Level Goal
Dependency tree / graph

Details

(Reporter)

Description

3 years ago
So, This might be filed in wrong place, but tonight I saw a handful of:


        _mysql_exceptions.OperationalError: (1040, 'Too many connections')

from a few masters.

I suspect this may be something we can alleviate somehow on the DB side.. it *could* also be related to many queries being blocked on a long running query or two, due to the nature of these DBs, but I wanted to file it before I forgot it.

Updated

3 years ago
Assignee: tmeyarivan → server-ops-database
Component: Operations : System Administration → Database Operations
HRm, we didn't get paged about that, so if there were too many connection it was transient.

We do need to build a new buildbot slave, though, so let's turn this into that.
Blocks: 1131628
I have built the new buildbot slave. Hopefully this helps.

However, I did check out variables and it looks like at some point in time, the master server hit max connections. Obviously adding a slave won't help...

But I also noticed that the master server is serving connections from buildbot_reader, too....shouldn't those be sent to the slave only?
(Reporter)

Comment 3

3 years ago
hit it on 12 seperate masters tonight (was ~3 or 4 yesterday) as well.

re: "But I also noticed that the master server is serving connections from buildbot_reader, too....shouldn't those be sent to the slave only?"  I forget the bug, but *you* made it part of the read-only pool as well, when we were load constrained.

So if we have a new slave now, we should drop it from the reader pool
(Reporter)

Comment 4

3 years ago
(In reply to Justin Wood (:Callek) from comment #3)
> hit it on 12 seperate masters tonight (was ~3 or 4 yesterday) as well.

Did I say 12, I meant something like ~24 (new mail flew in while I typed that, apparently google is delaying stuff from our relays by up to 10 min again)
OK, buildbot master is not in the read pool, 2 buildbot slaves are. There are no connections from buildbot_reader. :D

I think there's nothing left in this bug? Can we resolve?
(Reporter)

Comment 6

3 years ago
Will file a new if this hits again
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Updated

3 years ago
Blocks: 1141454
You need to log in before you can comment on or make changes to this bug.