808889 - [meta] prepare for the demise of zeus

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[Ryan Kelly [:rfkelly]]

Ops are planning to retire the current Zeus loadbalancer setup at some point in the future, and bring in an as-yet-undetermined replacement.  Some important aspects of our setup are currently handled within Zeus itself, such as database health checks (Bug 750964) and reg/node splitting (Bug 807839).

We need to prepare for the demise of Zeus by migrating any such responsibilities back into application code.  Anything that is currently handled by a trafficscript rule or other Zeus shenanigans needs to have a bug linked from here where we can discuss and implement a replacement.

:bobm and :atoll, I'll have to rely on you to identify what else needs to be done in this regard.

Comment 1

[:Atoll]

There are Zeus rules that I consider interesting that *might* need to be ported to another framework of some sort, either in the application or in the frontend servers:

- refuse requests with a URI longer than 3584 bytes (Sync add-on bug)
- return 503 for some or all requests
- add a Strict-Transport-Security header to all responses
- add X-Forwarded-{SSL,Protocol,Proto} headers to all requests
- add X-Forwarded-For headers to all requests
- redirect certain browserid.org and persona.org requests to other URIs
- rewrite the body of http 413 responses to be a valid JSON error
- rewrite http 499, 500, 502, 504 responses to http 503 responses with valid JSON
- block requests from certain IPs or usernames
- route node assignments to the node pool (soon deprecated)
- route verifier requests to a verifier pool (ineffective today)
- remove certain server headers from responses
- return http 401 or 503 for when sync backend nodes are migrating or downed (soon deprecated)
- add X-Weave-Backoff to some or all responses

A complete list of rules present on any Zeus server, stage or production, can be found under $ZEUS/zxtm/conf/rules/, and the list of *enabled* rules can be found either in the web UI or with "grep rule $ZEUS/zxtm/conf/vservers/*". ZEUS=/usr/local/zeus for all of our servers.