923571 - Please add nagios monitoring to vertica staging and production cluster

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Annie Elliott]

Today (10/3/13) Our production vertica experienced a segfault and was not usable. No one received any alerts, and we only figured this out because someone could not access it.

Please create nagios alerting for the two vertica clusters (staging an production) to monitor for this, along with any standard monitoring that is done for mysql instances.

:cyborgshadow helped clear the issue

Comment 1

[Sheeri Cabral [:sheeri]]

So this command shows how to monitor:

[[e-mail]3 ~]$ /opt/vertica/bin/admintools -t view_cluster DB      | Host | State 
---------+------+-------
 metrics | ALL  | UP

Comment 2

[Sheeri Cabral [:sheeri]]

Here's what it looks like when things are down:

[[e-mail]3 ~]$ /opt/vertica/bin/admintools -t view_cluster
 DB      | Host          | State 
---------+---------------+-------
 metrics | 192.168.100.7 | UP    
 metrics | 192.168.100.8 | UP    
 metrics | 192.168.100.9 | DOWN

Comment 3

[Sheeri Cabral [:sheeri]]

Also:

select * from vert_sys.v_catalog_nodes ;

Comment 4

[Matt Pressman [:mpressman]]

Added plugin check_vertica_cluster.sh
Committed revision 89747.

Comment 5

[Matt Pressman [:mpressman]]

This is completed

Comment 6

[Sheeri Cabral [:sheeri]]

Adding the check to puppet is not the last step in this. Once the nrpe check is added to puppet (in modules/nrpe/files/plugins/), we need to:

0) make sure the check is defined in nrpe (modules/nrpe/manifests/plugins.pp)
1) make sure the proper machines get the check (realize(Nrpe::Plugin['plugin_name']) in the puppet manifest
2) make sure the check is defined in Nagios (modules/nagios/manifests/mozilla/checkcommands.pp)
3) make sure the service is defined in Nagios (modules/nagios/manifests/mozillaservices.pp)
4) make sure the proper hosts *use* the check (modules/nagios/manifests/hosts/xxx.pp)

I've done all this in r94285. It may need tweaking, so I'll post here when the work is complete and the vertica cluster is actually being monitored.

For now, I've set it up to go to the postgres DBA oncall, so I didn't have to make a separate "vertica" group.

Comment 7

[Sheeri Cabral [:sheeri]]

I made a new "vertica-services" hostgroup too, in modules/nagios/manifests/mozilla/hostgroups.pp - this only needs to be done when there isn't already a hostgroup for the check.

Comment 8

[Sheeri Cabral [:sheeri]]

oh, there also needs to be a template in 

modules/nrpe/templates/nrpe.d

which I added.

Comment 9

[Sheeri Cabral [:sheeri]]

I got the script into nagios, and changed it so that if something is not UP it gives a warning. previously it checked for UP or DOWN, but the host could also be initializing.

I've put it on vertica[4,5,6].stage.metrics and put the check into a downtime window for a month. 

We need to figure out how to run the check as the dbadmin user, or give the nagios user permissions to run admintools.

Comment 10

[Sheeri Cabral [:sheeri]]

The script will also need to deal with the case that the db name isn't found - e.g. if there is no cluster running on the machine:

[[e-mail]3 ~]$ /opt/vertica/bin/adminTools -t view_cluster
 DB | Host | State 
----+------+-------

[[e-mail]3 ~]$

Comment 12

[Sheeri Cabral [:sheeri]]

This has been added to puppet.

Comment 13

[Sheeri Cabral [:sheeri]]

This is done, except for what's in bug 1110972.