792042 - audit dns and nagios for ad.mozilla.com and releng.ad.mozilla.com domain controllers

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Amy Rich [:arr] [:arich]]

I noticed that there were a number of missing/misconfigured nagios bits for the domain controllers and then dug deeper and also found missing DNS information for them as well.  Need to do an audit to make sure that all of the correct information is in place.

Comment 1

[Amy Rich [:arr] [:arich]]

Fixed:

* missing PTR records for dc3 and dc8
* nagios for dc1 incorrectly part of the config at admin1.infra.scl1.mozilla.com.  moved to scl3 nagios config in puppet.
* no nagios service checks for dc8, added.

Still needs fixing:
* no nagios checks for dc3.  added to scl3 nagios config in puppet, but all but ping are failing.  likely nrpe is not installed or is not allowed through the host or router firewall.
* no nagios service checks for dc9.  added, but windows services failing.
* all hosts are missing NIC information in inventory some may be labeled building when they're actually production.
* missing SRV records for dc4
* no nagios checks for dc4, added to phx1, but all are failing including ping. likely nrpe is not installed or is not allowed through the host or router firewall and the host is down.

Is dc4 installed at all?

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Amy Rich [:arich] [:arr] from comment #1)
> * no nagios checks for dc3.  added to scl3 nagios config in puppet, but all
> but ping are failing.  likely nrpe is not installed or is not allowed
> through the host or router firewall.

I installed nsclient++ on dc3, and nc -vz works fine on port 5666 when it's running (and not when it's not running), but it's not giving data back to nagios:
  CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages

> * missing SRV records for dc4
> * no nagios checks for dc4, added to phx1, but all are failing including
> ping. likely nrpe is not installed or is not allowed through the host or
> router firewall and the host is down.
> 
> Is dc4 installed at all?

Nope, AFAIK.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Opening the firewall completely on dc3 didn't help.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

allowed_hosts problem:
  https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=27918823

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Amy Rich [:arich] [:arr] from comment #1)
> * no nagios service checks for dc9.  added, but windows services failing.

I removed the "Windows Services" check, since IMHO it's bogus and even if it worked it wouldn't tell us much of use.  See bug 792464.