761675 - Check and possibly fix RabbitMQ stats in Ganglia (at least PHX1)

Status: RESOLVED WORKSFORME

(mozilla.org Graveyard :: Server Operations, task)

Description

[Brandon Burton [:solarce]]

In reviewing https://ganglia-phx1.mozilla.org/ganglia/?c=addons-dev&h=celery-dev1&m=cpu_report&r=hour&s=descending&hc=4&mc=2#metric_rabbit_dev_builder_celery_messages_unacknowledged :ashish and I both noted that a number of the RabbitMQ metrics are empty, this may indicate an issue

Please review the RabbitMQ stats and confirm they are being populated correctly or fix as necessary

Comment 1

[Brandon Burton [:solarce]]

Just wanted to see if this could be looked at soon, as these stats are useful for troubleshooting and capacity planning

Comment 2

[Ashish Vijayaram [:ashish]]

I looked into this earlier today and am still clueless about how/where these metrics are sourced from. There were some pointers to a php file for prod but little else.

CC'ing oremj for halp!

Comment 3

[Ashish Vijayaram [:ashish]]

So, all these graphs are working as intended. I verified the data by running rabbitmqctl and the graphs that are "blank" are indeed only 0. Also, bad/no data would have -nan- instead.