1161894 - Increase fhr.data.mozilla.com to 3 traffic IPs

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[:Atoll]

Currently, fhr.data.mozilla.com has only a single traffic IP. However in adverse conditions, the (valid and reasonable) retry/backoff logic in the FHR client can lead to an increase of up to 2x-3x the normal connection rate. We can handle that volume of SSL negotiations, but not on a single load balancer.

I propose we increase fhr to 3 traffic IPs, so that a drastic increase in FHR traffic for whatever reason does not trigger a negative feedback loop with ever-increasing client retries due to saturation. We're not at risk for outbound bandwidth saturation, just need more CPUs assigned to handle SSL negotiation if it fails.

Comment 1

[:Atoll]

We discussed this and don't see a need to increase capacity at this time. A full rewrite of FHR is coming up over the next 6-18 weeks (depending on release schedule) and we have mitigation strategies in place if any further incidents occurred.