710826 - ganglia thinks signing{1,2}.build.scl1.mozilla.com are offline

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Chris AtLee [:catlee]]

http://ganglia1.build.scl1.mozilla.com/ganglia/?c=RelEngSCL1&h=signing1.build.scl1.mozilla.com&m=load_one&r=hour&s=descending&hc=4&mc=2

http://ganglia1.build.scl1.mozilla.com/ganglia/?c=RelEngSCL1&h=signing2.build.scl1.mozilla.com&m=load_one&r=hour&s=descending&hc=4&mc=2

These used to work, not sure what changed.

Comment 1

[Chris AtLee [:catlee]]

sorry, I should have added that these machines are up and healthy, it's just ganglia that's confused.

Comment 2

[Amy Rich [:arr] [:arich]]

If I recall correctly, bhearsum added iptables filtering to these machines.  Quite possibly that's what broke ganglia.  If you turn off the filtering, do they come back (I don't want to do so since I don't want to break/interrupt whatever the filtering was set up for)?

Comment 3

[Amy Rich [:arr] [:arich]]

Based on the fact that releng-mirror01 was also showing offline, I went digging and found that ganglia1 had the wrong netmask set.  I corrected it and restarted gexecd and gmond on all of the feeder servers and the servers that were missing and everything appears to be back to normal now.