1093930 - Return 503 + RetryAfter header for transient Token Server failures

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Server: Token, defect)

Description

[Bob Micheletto [:bobm]]

Catch transient failures, such as an exhausted DB connection pool, from upstream sources and return a 503 response with a sensible RetryAfter header.  A sliding RetryAfter isn't strictly necessary, but some fuzzing around the timer is desirable.

The purpose is to gently back off and rebalance incoming traffic in scenarios where the service is temporarily overwhelmed, and to stop Fx clients from displaying the yellow error bar in these situations.

It is plausible to address these events at the webserver front-end, in this case Openresty.  However, in the interest of keeping application logic in the application itself, we should first consider the possibilities of putting it there.

Comment 1

[Ryan Kelly [:rfkelly]]

For the particular case of db-related errors, we should really pull the connection-pool-handling stuff out of sync and put it in a shared util library for tokenserver to use.

Comment 2

[Bob Micheletto [:bobm]]

The Nginx configuration for Tokenserver is doing this now.  Closing this bug.