693511 - [some?] tegra nagios alerts seem to be notifying immediately rather than waiting

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Aki Sasaki (not active)]

There's a decent amount of:

* tegra XXX PING is CRITICAL
* 10-20 minutes later, tegra XXX PING is OK.

Ooc, what are these set to?  They're 12/12 failed before notifying, but I don't remember what the interval is between those checks.

The minis appear to check 30/30 before notifying; if the intervals are the same, we might want to increase the time-before-notify on the tegras as well.

Comment 1

[Amy Rich [:arr] [:arich]]

There was an issue with the script that generates the hosts files that was ignoring the first_notification_delay for the PING checks (host checks and tegra-tcp checks were fine).  I've fixed the script and remade the config files.  The 720 minute delay should now be in effect.

Comment 2

[Amy Rich [:arr] [:arich]]

There was an issue with the script that generates the hosts files that was ignoring the first_notification_delay for the PING checks (host checks and tegra-tcp checks were fine).  I've fixed the script and remade the config files.  The 720 minute delay should now be in effect.