615547 - Move Sphinx load balancing to Zeus

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[James Socol [:jsocol, :james]]

Keeping with the trend of moving load balancing to Zeus, it seems like we should consider moving Sphinx to the ZLB as well.

We see a slow but steady stream of network errors from Sphinx that look roughly like:

>  File "/data/www/support.mozilla.com/kitsune/apps/search/sphinxapi.py", line 223, in _GetResponse
>    (status, ver, length) = unpack('>2HL', sock.recv(8))
>
> error: unpack requires a string argument of length 8

I'm not sure of Zeus instead of Netscaler/ACE would alleviate these or not.

So my questions are:
1) Can we make Sphinx work with the ZLB?
2) If so, should we?
3) If so, can we?

Comment 1

[matthew zeier [:mrz]]

Not sure if I have the dep right.  This will have to happen when SUMO moves to PHX1 anyways and then SJC1 can be "rebuilt" to match.  

At that point, it won't be a downtime event.

Comment 2

[Jeremy Orem [:oremj]]

This is ready in PHX. I also added the ip as sphinx-sumo in puppet:

+        "sphinx-sumo":
+            ip => $fqdn ? {
+                /^pp-/ => "10.8.83.69",
+                /phx1.mozilla.com$/ => "10.8.83.69",
+                default => "10.2.83.69",
+            };