1309897 - clean up monitoring

Status: RESOLVED INCOMPLETE

(Webtools Graveyard :: DXR, defect)

Description

[Kendall Libby [:fubar] (he/him)]

off the top of my head:

- new blades need to have standard host monitoring enabled
- ensure all nodes have production monitoring (not -preprod)
- remove monitoring for hosts that have been pulled from the ES cluster
- create ES cluster check for health, instead of per-node (mmm, alert spam)
- verify we're monitoring jenkins

it might be useful to dig up the old MXR process checks for long indexing jobs and repurpose to look for wedged indexing jobs (eg would catch builds that segfault and never finish)

Comment 1

[Amy Rich [:arr] [:arich]]

For nodes that are esx vms, the entry in sysadmins/puppet/trunk/modules/nagios/manifests/hosts/scl3.pp should look like:

        '<FQDN>' => {
            parents => 'esx-cluster1.ops.scl3.mozilla.com',
            hostgroups => [
                'generic',
                'virtual',
            ]     
        }, 

Hosts may have other hostgroups specified depending on their function (e.g. 'dxr-jenkins-slaves', )

For hardware, that should be more like (this assumes they're HP machines):

        '<FQDN>' => {
            parents => '<look up the switch it's connected to in inventory>',
            hostgroups => [
                'generic',
                'hp-servers',
            ]     
        }, 

We may also want to set/override the contact_groups setting if we're setting up cluster checks (so that people don't get paged for one node being down). I'll leave this to fubar to help explain/define.

Creating a cluster check would be done in: sysadmins/puppet/trunk/modules/nagios/manifests/mozilla/clusterchecks.pp


Not sure what we need to be monitoring for jenkins, so I'll leave that one up to fubar as well.

Comment 2

[:dhouse]

assigning to myself