520594 - moz2-darwin* and bm-xserve machines having trouble with nagios since puppet rollout

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

It seems that they take 10-15 minutes to start the nrpe daemon, which sets off quite a few alerts. Puppet does manage the nagios config / daemon, so it's possible that it busted it in some way.

Comment 2

[[e-mail] (:bhearsum)]

I'll look into this a bit.

Comment 3

[[e-mail] (:bhearsum)]

Attachment: Manage the nrpe plist file with Puppet

Turns out we're not managing this file yet, this will get us doing that. Incoming, is a copy of the plist file.

Comment 4

[Chris AtLee [:catlee]]

Comment on attachment 404659 [details] [diff] [review]
Manage the nrpe plist file with Puppet


>+          source => "${fileroot}darwni9/nrpe.plist",

Looks good, as long as you fix the typo.

Comment 5

[[e-mail] (:bhearsum)]

Attachment: nrpe plist

Here's the error I found in the system log:
Oct  5 11:44:05 localhost com.apple.launchd[1] (org.nagios.nrpe): Unknown key: ServiceDescription
Oct  5 11:44:58 moz2-darwin9-slave16 com.apple.launchd[1] (org.nagios.nrpe): Unknown key: ServiceDescription

This is a copy of the plist file without that key, which seems to work right away, rather than 20 minutes later.

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 404659 [details] [diff] [review]
Manage the nrpe plist file with Puppet

changeset:   59:4e22d906aa0b

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 404660 [details] [diff] [review]
nrpe plist

Checking in nrpe.plist;
/mofo/puppet-files/darwin9/nrpe.plist,v  <--  nrpe.plist
initial revision: 1.1
done

Comment 8

[[e-mail] (:bhearsum)]

I updated the Puppet masters, these should be OK now. Leaving this bug open for now.

Comment 9

[[e-mail] (:bhearsum)]

This didn't fix it, apparently. I've disabled the notifications for now, to quiet down #build. I'll come back to this a bit later today.

Comment 10

[[e-mail] (:bhearsum)]

Attachment: only run enablenrpe if nrpe.plist has changed

This patch makes Puppet not exec enablerpe unless:
* /Library/LaunchDaemons/nrpe.plist doesn't exist (this is the creates => part)
* setup-nrpe gets run (this is the first part of the subscribe)
* /usr/local/nagios/etc/nrpe.plist changes (the second part of the subscribe)

I tested a few reboot cycles with and without this patch and the results were 100% consistent:
Without the patch, the nagios server gets connection refused right after boot
With the patch, it is able to properly execute its checks immediately after boot.

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 404882 [details] [diff] [review]
only run enablenrpe if nrpe.plist has changed

changeset:   62:758b10a407d1

Comment 12

[[e-mail] (:bhearsum)]

I haven't seen any more of these fail, and I explicitly saw one work immediately after booting. I think the last patch did it.