997231 - Need alert at 80% of max DB connections for Firefox Accounts/Sync

Status: VERIFIED FIXED

(Cloud Services :: Operations: Metrics/Monitoring, task)

Description

[Karl Thiessen [:kthiessen, he/him]]

Per :jrgm, exceeding the number of DB connections available in FxA/Sync is catastrophic.  So we'd like to know when we're getting close to the red zone.

CC'ing jrgm, Edwin, and Gene, who are likely to have any details needed to make this happen.

Comment 1

[Karl Thiessen [:kthiessen, he/him]]

It's been 60 days.  Any word on this?

Comment 2

[Karl Thiessen [:kthiessen, he/him]]

Escalating this to the ops people who are likely to be most in the line of fire: :bobm, :whd.  If there are other folks who would know about this, please bring them in via cc or needinfo.

Bob, Wes, what do we have in terms of monitoring?

Comment 3

[Bob Micheletto [:bobm]]

(In reply to Karl Thiessen [:kthiessen] from comment #2)
> Escalating this to the ops people who are likely to be most in the line of
> fire: :bobm, :whd.  If there are other folks who would know about this,
> please bring them in via cc or needinfo.

Is this a monitoring request for max connections to the RDS instance for FxA?  The max connections to the Token DB? Or, the individual Sync servers?

> Bob, Wes, what do we have in terms of monitoring?

On individual Sync servers the Max connections to the DBs should not be hit, but rather get queued / throttled at the application layer where worse case it can result in an error that should be handled gracefully by the client.  We have the ability to detect when this happens, and can appropriately deal with the situation from a capacity perspective.

Comment 4

[Karl Thiessen [:kthiessen, he/him]]

(In reply to Bob Micheletto [:bobm] from comment #3)
> (In reply to Karl Thiessen [:kthiessen] from comment #2)
> > Escalating this to the ops people who are likely to be most in the line of
> > fire: :bobm, :whd.  If there are other folks who would know about this,
> > please bring them in via cc or needinfo.
> 
> Is this a monitoring request for max connections to the RDS instance for
> FxA?  The max connections to the Token DB? Or, the individual Sync servers?

I'm most concerned about the first two, as the Sync Servers have some application-layer backoff, as you state.  :jrgm will likely have more details.

Comment 5

[Bob Micheletto [:bobm]]

Request for Max Connections monitor for Token and FxA RDS.

Comment 6

[Benson Wong [:mostlygeek]]

Added to stack driver. 

The max DB connections is: 1232
Obtained with the SQL: show variables like '%conn%' 

I set a stackdriver alert for when the connections go above 900. Currently with 6 servers we're at ~250 connections. So we're good to scale up to the max servers in the Autoscale group (21).

Comment 7

[Karl Thiessen [:kthiessen, he/him]]

Very cool.  Thank you, Ben.

Comment 8

[Karl Thiessen [:kthiessen, he/him]]

I will take the action to check in with :jrgm next week in Mt View and make sure that this covers everything he needed/wanted, after which I will mark this bug VERIFIED.

Comment 9

[John Morrison [:jrgm]]

Seems good to me. Thanks.

Comment 10

[Karl Thiessen [:kthiessen, he/him]]

Marking VERIFIED per comment 8.