627870 - New nagios checks for Socorro

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Laura Thomson :laura]

This is the list we came up with in the meeting:
- mware check to load a crash from hbase
- zeus health checks (check that a back end server is down via the web interface)
- no new report added to PG in x time
- no update to tcbs tcbu  in PG
- monitor all logs for ERROR/CRITICAL

Comment 1

[Justin Dow [:jabba]]

Also need to add this:

define command{
        command_name    check_postgres_partition_check
        command_line    $USER1$/custom/check_postgres.pl --action=custom_query --valtype=integer -w 1 -c 2 --query="select * from check_partitions(ARRAY['reports','extensions','plugins_reports','frames'],1);" --dbname breakpad -u nagiosdaemon
}

define service{
        use                             postgresql-service
        hostgroups                                     pgmaster-servers
        service_description             PostgreSQL Breakpad Partition Check
        check_command                   check_postgres_partition_check
        check_interval                  1440
}

Comment 2

[Justin Dow [:jabba]]

And this:


diff --git a/clients/mozilla/nagios/mozilla-db-servers.cfg b/clients/mozilla/nagios/mozilla-db-servers.cfg
index ca247e0..395e720 100644
--- a/clients/mozilla/nagios/mozilla-db-servers.cfg
+++ b/clients/mozilla/nagios/mozilla-db-servers.cfg
@@ -55,11 +55,6 @@ define command{
 }
 
 define command{
-        command_name    check_postgres_idletxn_count
-        command_line    $USER1$/custom/check_postgres.pl -H $HOSTADDRESS$ -u nagiosdaemon --action=txn_idle --warning=$ARG1$ --warning-count=$ARG2$ --critical=$ARG3$ --critical-count=$ARG4$
-}
-
-define command{
         command_name    check_postgres_backends
         command_line    $USER1$/custom/check_postgres.pl -H $HOSTADDRESS$ -u nagiosdaemon --action=backends --warning=$ARG1$ --critical=$ARG2$
 }
@@ -199,17 +194,18 @@ define service{
 #        check_command                   check_postgres_last_vacuum!1d!7d
 #}
 #
-#define service{
-#        use                             postgresql-service
-#        hostgroups                                    pgmaster-servers
-#        service_description             PostgreSQL Breakpad Bloat
-#        check_command                   check_postgres_bloat!breakpad!400%!1000%
-#        check_interval                                        60
-#}
 # END FIXME: Josh wants to update these checks in check_postgres
 
 define service{
         use                             postgresql-service
+        hostgroups                                     pgmaster-servers
+        service_description             PostgreSQL Breakpad Bloat
+        check_command                   check_postgres_bloat!breakpad!50MB and 200%!500MB and 1000%
+        check_interval                                         60
+}
+
+define service{
+        use                             postgresql-service
         hostgroups                                     pg-servers
         service_description             PostgreSQL Query Time
         check_command                   check_postgres_query_time!5m!45m
@@ -227,7 +223,7 @@ define service{
         use                             postgresql-service
         hostgroups                                     pgmaster-servers
         service_description             PostgreSQL Idle Transactions Count
-        check_command                   check_postgres_idletxn!1m!10!1h!10
+        check_command                   check_postgres_idletxn!10 for 1m!10 for 1h
 }
 
 define service{


This is just so I don't forget. Jeff Frost sent me these in an e-mail...

Comment 3

[[:jberkus] Josh Berkus]

Note from Josh to himself:  we need a new last_record check on reports_duplicates.

Comment 4

[Justin Dow [:jabba]]

I've implemented the last of the postgres checks. I think really still outstanding is the status.mozilla.net setup in bug 638421. I've implemented the zeus checks for all services behind zeus and I feel quite confident that at this point we have sufficient monitoring to know when there is a problem and sufficient for oncall to be alerted and take the proper action. Any further monitors should each get their own bug so that it is easier to track what is done and what isn't.