510755 - Enable notification/page for host flapping in Nagios

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[David Tran]

"Whenever a service check is performed that results in a hard state or a soft recovery state, Nagios checks to see if the services has started or stopped flapping. It does this by storing the results of the last 21 checks of the service in an array."

We need to have Nagios page when a host starts flapping and when it starts - otherwise, we'll occasionally see lots of CRITICAL pages without any corresponding recovery messages.

Comment 1

[Corey Shields [:cshields]]

08:16 <@justdave> something's messed up with our config that we haven't figured out
08:17 <@justdave> if something's paged critical and then nagios decides it's flapping, nagios will stop sending pages of any kind
08:17 <@justdave> it's supposed to send you another page saying "I've entered flap detection mode for this service"
08:17 <@justdave> THAT's what's not happening

Try to reproduce and fix this..

Comment 2

[Rob Tucker [:rtucker]]

I think that we're better off just disabling flap detection. 

If we have nagios send an alert that says it's entered flap detection, this notification will inevitably just get forgotten about anyway.

I'm going to disable flap detection and see if it causes any issues, but I doubt that it will.

Comment 3

[Rob Tucker [:rtucker]]

I've disabled flap detection in the Nagios CS servers. They are the hosts that determine if a service is flapping, since the other servers are just doing passive checks initiated by the CS.

This should resolve the issue. I'm not sure of a good way to test this however.

Comment 4

[Rob Tucker [:rtucker]]

It doesn't look like disabling flap detection caused any issues. I did not think that it would. I'm going to close this out.