761663 - Add generic celery hosts to PHX1 Ganglia

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Brandon Burton [:solarce]]

Please add the following hosts to PHX1 Ganglia, host stats, rabbitmq stats, and celeryd stats needed, https://ganglia-phx1.mozilla.org/ganglia/?c=addons-dev&h=celery-dev1&m=cpu_report&r=hour&s=descending&hc=4&mc=2#metric_rabbit_dev_builder_celery_messages_unacknowledged is a good example of what kinds of stats are needed

Hosts:

generic-celery1.dev.seamicro.phx1.mozilla.com
generic-celery1.stage.seamicro.phx1.mozilla.com
generic-celery1.seamicro.phx1.mozilla.com
generic-celery2.seamicro.phx1.mozilla.com

Comment 1

[Brandon Burton [:solarce]]

Just wanted to see if this could be looked at soon, as these stats are useful for troubleshooting and capacity planning

Comment 2

[Ashish Vijayaram [:ashish]]

So I setup the multicast IPs for this on Fri to show up at https://ganglia-phx1.mozilla.org/ganglia/?c=generic-dev and https://ganglia-phx1.mozilla.org/ganglia/?c=generic-web

generic-dev has 1 host named "generic-celery-dev" but the rest of the hosts don't show up. Shall troubleshoot this further.

Comment 3

[Ashish Vijayaram [:ashish]]

I haven't been able to looking into this much given ganglia-phx's general instability. From what I gather, this needs a mcast + ucast setup between the nodes.

Comment 4

[Ashish Vijayaram [:ashish]]

This took a while to sort out. Had to clean up IP addresses of old ganglia hosts in gmond templates. Also, puppet isn't running on any of these servers. I've setup the gmond configs on the servers by hand.

https://ganglia-phx1.mozilla.org/ganglia/?c=generic-celery