675052 - graphite.dmz.sjc1 is dead

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Jeff Balogh (:jbalogh)]

oremj can't ssh in and https://graphite-sjc.mozilla.org/ is returning the zeus error page.

This isn't critical to amo running but it makes me uncomfortable to not have data on how our pages are doing.

After this comes back up we should get nagios tracking it and try to figure out what took it down. It died right after today's amo push but I don't know think we started sending it significantly more traffic.

Comment 1

[Justin Dow [:jabba]]

phong tripled the amount of memory in this VM, and I've added nagios monitoring to it. Perhaps we are at a point where we should consider a physical box for arecibo (oremj mentioned it was httpd getting slammed)... This box hosts arecibo, ganglia and graphite. I think perhaps we might want to split each of those services onto their own physical boxes and use this host (and its counterpart in phx) as a reverse-proxy for those services, so that we end up with a single entry point for monitoring, but one service doesn't kill the others... Will come up with a plan in the next few days for this.

Comment 2

[Jeff Balogh (:jbalogh)]

Thanks. I'll be happy to see Arecibo move off of this box so we don't bring down our monitoring whenever the site has some tracebacks.

Arecibo went crazy because we pushed new code that was crashing (until we switched it off with a flag), and Arecibo couldn't keep up with the traceback traffic.

Comment 3

[Andy McKay]

I'll have a check through the server, sure i can optimize or throttle the server. 

Further, i'll implement some sort of throttling in the client libs. There's just no point in sending that amount of traffic to Arecibo, if its just the same error over and over again.