1253079 - Please review temperature monitoring

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Service Requests, task)

Description

[James Barnell [:jbarnell]]

Team,

BER2 had a temperature problem on the SRX in the IDF.  We did not see any alerting via IRC or pagerduty.  I'd like to understand the monitors currently in place for this office (ideally all).  We'd also like to begin monitoring temperature of all devices where there is support (Juniper definitely does this)

Please let me know what you need from me on this.

Comment 1

[Peter Radcliffe [:pir]]

(In reply to James Barnell [:jbarnell] from comment #0)
> BER2 had a temperature problem on the SRX in the IDF.  We did not see any
> alerting via IRC or pagerduty.

> I'd like to understand the monitors currently in place for this office (ideally all).

puppet/trunk/modules/nagios/manifests/hosts/scl3.pp
puppet/trunk/modules/nagios/manifests/mozilla/services.pp


There wasn't any Juniper temperature monitoring for ber2. It was never added to the list when the office was set up.

There are no hostgroups that the firewall devices are in by default:
        'fw1.ops.ber1.mozilla.net' => {
            normal_check_interval => '1',
            retry_check_interval => '1',
            max_check_attempts => '2'
        },

        'fw1.ops.ber2.mozilla.net' => {
            normal_check_interval => '1',
            retry_check_interval => '1',
            max_check_attempts => '2'
        },

So I can't just apply it to all fw* devices currently. There's also the matter of different SNMP community strings between devices which made it hard to automatically add this monitoring, though I think after the gradual standardisation work none of the fw boxes we're monitoring for temperature currently are using alt strings any more.

I'll create a new hostgroup 'juniper-temperature' and apply it to all the firewlls so new offices should get the monitoring automatically as long as they also get added to that hostgroup. If you want to add more than just firewalls to this group then the oncall docs will need expanding with details on how to handle and escalate these alerts since the current docs are focused almost entirely on an office/IDC overheating not general hardware.


The UPS temperature probe config for ber2 was copied from ber1 which has an exception added for high temperatures because ber1 doesn't have a real IDC and it kept overheating. I've just fixed ber2 back to normal levels but even so the old config whould have reached warning at 40C and critical at 45C. How warm did the room get?  With a normal config it has started alerting at 37C which is only just above warning level (35C).


Juniper firewall monitoring and UPS probe monitoring should be in place for all offices and all UPS units that we are made aware of having the probes.

> We'd also like to begin monitoring temperature of all devices where there is support (Juniper
> definitely does this)

Yes, Juniper does and many other bits of networking gear do via SNMP. We're monitoring temperature from Juniper firewall boxes at the moment.

We have a lot of Juniper and other switches/network gear that will support temperature monitoring. Adding them all would be a non-trivial scaling/load addition to Nagios and a lot more noise when offices drop on/off the network (which is already very noisy for us). Different IDCs run at different temperatures (as we've already seen with ber1) so it would likely require a fairly complicated config set to not flap all the time but still give useful alerting. I can't say I feel this would be a good thing.

Why do you want to monitor the temperature of all devices not just a selection per office or IDC?

Comment 2

[Peter Radcliffe [:pir]]

Index: puppet/trunk/modules/nagios/manifests/hosts/pek1.pp
===================================================================
--- puppet/trunk/modules/nagios/manifests/hosts/pek1.pp	(revision 115384)
+++ puppet/trunk/modules/nagios/manifests/hosts/pek1.pp	(working copy)
@@ -6,7 +6,10 @@
     $switches = {
         'fw1.ops.pek1.mozilla.net' => {
             parents => 'admin1.private.pek1.mozilla.com',
-            contact_groups => 'sysalerts, netopsalertslist'
+            contact_groups => 'sysalerts, netopsalertslist',
+            hostgroups => [
+                'juniper-temperature',
+            ]
         },
          'fw1.corp.pek2.mozilla.net' => {
 	    #parents => 'admin1.private.pek1.mozilla.com',
Index: puppet/trunk/modules/nagios/manifests/hosts/scl3.pp
===================================================================
--- puppet/trunk/modules/nagios/manifests/hosts/scl3.pp	(revision 115385)
+++ puppet/trunk/modules/nagios/manifests/hosts/scl3.pp	(working copy)
@@ -941,7 +941,8 @@
             max_check_attempts => '2',
             parents => 'fw1.scl3.mozilla.net',
             hostgroups => [
-                'scl3-fw'
+                'scl3-fw',
+                'juniper-temperature',
             ]
         },
 
@@ -1012,6 +1013,7 @@
             max_check_attempts => '2',
             hostgroups => [
                 'office-mtv2',
+                'juniper-temperature',
             ]
         },
         'fw1.console.mtv2.mozilla.net' => {
@@ -2524,6 +2526,7 @@
             max_check_attempts => '2',
             hostgroups => [
                 'office-sfo1',
+                'juniper-temperature',
             ]
         },
         'admin1a.private.sfo1.mozilla.com' => {
@@ -3137,7 +3140,10 @@
             normal_check_interval => '1',
             retry_check_interval => '1',
             max_check_attempts => '2',
-            parents => 'fw1.scl3.mozilla.net'
+            parents => 'fw1.scl3.mozilla.net',
+            hostgroups => [
+                'juniper-temperature',
+            ]
         },
         'admin1a.private.pdx1.mozilla.com' => {
             parents                        => 'fw1.ops.pdx1.mozilla.net',
@@ -3325,6 +3331,7 @@
             max_check_attempts => '2',
             hostgroups => [
                 'office-yvr1',
+                'juniper-temperature',
             ]
         },
         'admin1.yvr1.mozilla.com' => {
@@ -3674,6 +3681,7 @@
             contact_groups => 'sysalerts, netopsalertslist',
             hostgroups => [
                 'office-par1',
+                'juniper-temperature',
             ]
         },
         'fw1-0.console.par1.mozilla.net' => {
@@ -4132,7 +4140,10 @@
             normal_check_interval => '1',
             retry_check_interval => '1',
             max_check_attempts => '2',
-            contact_groups => 'sysalerts, netopsalertslist'
+            contact_groups => 'sysalerts, netopsalertslist',
+            hostgroups => [
+                'juniper-temperature',
+            ]
         },
         'fw1.private.lon1.mozilla.net' => {
             normal_check_interval => '1',
@@ -4354,7 +4365,10 @@
             contact_groups => 'sysalerts, netopsalertslist',
             normal_check_interval => '1',
             retry_check_interval => '1',
-            max_check_attempts => '2'
+            max_check_attempts => '2',
+            hostgroups => [
+                'juniper-temperature',
+            ]
         },
         'admin1.private.tor1.mozilla.com' => {
             normal_check_interval => '1',
@@ -4528,7 +4542,10 @@
             parents                       => 'fw1.tier2.akl1.mozilla.net, fw1.tier3.akl1.mozilla.net',
             normal_check_interval         => '1',
             retry_check_interval          => '1',
-            max_check_attempts            => '2'
+            max_check_attempts            => '2',
+            hostgroups => [
+                'juniper-temperature',
+            ]
         },
         'fw1-0.console.akl1.mozilla.net' => {
             normal_check_interval => '1',
@@ -4650,7 +4667,10 @@
         'fw1.ops.ber1.mozilla.net' => {
             normal_check_interval => '1',
             retry_check_interval => '1',
-            max_check_attempts => '2'
+            max_check_attempts => '2',
+            hostgroups => [
+                'juniper-temperature',
+            ]
         },
         'admin1.private.ber1.mozilla.com' => {
         # admin1.ber1 doesn't run openvpn
@@ -4749,6 +4769,9 @@
             retry_check_interval => '1',
             max_check_attempts => '2'
         },
+        # Temperature probe levels are higher than normal for BER1 because
+        # the room is poorly cooled. DO NOT COPY THIS CONFIG to another
+        # datacenter unless it is also known to be too warm.
         'ups1.r201-1.ops.ber1.mozilla.net' => {
             parents => 'fw1.ops.par1.mozilla.net',
             contact_groups => 'netopsalertslist',
@@ -4778,7 +4801,10 @@
         'fw1.ops.ber2.mozilla.net' => {
             normal_check_interval => '1',
             retry_check_interval => '1',
-            max_check_attempts => '2'
+            max_check_attempts => '2',
+            hostgroups => [
+                'juniper-temperature',
+            ]
         },
         'fw1.tier1.ber2.mozilla.net' => {
             contact_groups => 'sysalertsonly'
@@ -4999,7 +5025,10 @@
             normal_check_interval => '1',
             retry_check_interval => '1',
             max_check_attempts => '2',
-            parents => 'fw1.scl3.mozilla.net'
+            parents => 'fw1.scl3.mozilla.net',
+            hostgroups => [
+                'juniper-temperature',
+            ]
         },
         'admin1a.private.tpe1.mozilla.com' => {
             parents                        => 'fw1.ops.tpe1.mozilla.net',
Index: puppet/trunk/modules/nagios/manifests/mozilla/hostgroups.pp
===================================================================
--- puppet/trunk/modules/nagios/manifests/mozilla/hostgroups.pp	(revision 115384)
+++ puppet/trunk/modules/nagios/manifests/mozilla/hostgroups.pp	(working copy)
@@ -661,6 +661,9 @@
         'ups-temperature-probe-high' => {
             alias => "UPS with an external temperature probe in a warm room",
         },
+        'juniper-temperature' => {
+            alias => "Juniper temperature monitoring via SNMP",
+        },
         'lightcontrol-mtv2' => {
             alias => "Mountain View Light Control",
         },
Index: puppet/trunk/modules/nagios/manifests/mozilla/services.pp
===================================================================
--- puppet/trunk/modules/nagios/manifests/mozilla/services.pp	(revision 115384)
+++ puppet/trunk/modules/nagios/manifests/mozilla/services.pp	(working copy)
@@ -7393,52 +7393,17 @@
             contact_groups => 'sysalerts',
             normal_check_interval => 10,
             notification_interval => 30,
-            host_name => $::fqdn ? {
+            hostgroups => $::fqdn ? {
                 /^nagios(1|2).private.scl3.mozilla.com$/ => [
-                    'fw1.ops.mtv2.mozilla.net',
-                    'fw1.ops.lon1.mozilla.net',
-                    'fw1.ops.par1.mozilla.net',
-                    'fw1.ops.tor1.mozilla.net',
-                    'fw1.ops.tpe1.mozilla.net',
-                    'fw1.ops.yvr1.mozilla.net',
-                    'fw1.ops.akl1.mozilla.net',
-                    'fw1.ops.pdx1.mozilla.net',
-                    'fw1.ops.sfo1.mozilla.net',
+                    'juniper-temperature',
                 ],
                 'nagios1.private.pek1.mozilla.com' => [
-                    'fw1.ops.pek1.mozilla.net',
+                    'juniper-temperature',
                 ],
                 default => [
                 ]
             },
-            hostgroups => $::fqdn ? {
-                default => [
-                ]
-            },
         },
-        #        'check_juniper_temp_office' => {
-        #    service_description => "Juniper temperature",
-     # The warning and critical values are comma seperated lists
-     # that apply in order to the comma seperate list of regular
-     # expressions the check searches for in the next option.
-     # This version uses the office snmp readonly string.
-     #    check_command => "check_snmp_temp!juniper!50,50!55,55!node0.Routing.Engine,node1.Routing.Engine!$snmp_readonly_office",
-     #       contact_groups => 'sysalerts',
-     #       normal_check_interval => 10,
-     #       notification_interval => 30,
-     #       host_name => $::fqdn ? {
-     #           /^nagios(1|2).private.scl3.mozilla.com$/ => [
-     #               'fw1.ops.pdx1.mozilla.net',
-     #               #'fw1.ops.sfo1.mozilla.net',
-     #           ],
-     #           default => [
-     #           ]
-     #       },
-     #       hostgroups => $::fqdn ? {
-     #           default => [
-     #           ]
-     #       },
-     #   },
         'check_apc_ext_temp' => {
             service_description => "APC UPS external temperature",
             check_command => "check_snmp_temp!apc-ext!35!40!External!$snmp_readonly",
@@ -7455,7 +7420,7 @@
         },
         'check_apc_ext_temp_high' => {
             service_description => "APC UPS external temperature",
-            check_command => "check_snmp_temp!apc-ext!40!45!External!$snmp_readonly",
+            check_command => "check_snmp_temp!apc-ext!38!42!External!$snmp_readonly",
             contact_groups => 'sysalerts',
             normal_check_interval => 10,
             notification_interval => 30,


pir@disapp> svn ci -m "rework juniper temperature monitoring to use hostgroups, bug 253079"
Sending        puppet/trunk/modules/nagios/manifests/hosts/pek1.pp
Sending        puppet/trunk/modules/nagios/manifests/hosts/scl3.pp
Sending        puppet/trunk/modules/nagios/manifests/mozilla/hostgroups.pp
Sending        puppet/trunk/modules/nagios/manifests/mozilla/services.pp
Transmitting file data ....done
Committing transaction...
Committed revision 115386.

Comment 3

[Peter Radcliffe [:pir]]

This page has a list of all the hosts that currently have Juniper temperature monitoring configured:
https://nagios.mozilla.org/scl3/cgi-bin/status.cgi?hostgroup=juniper-temperature&style=detail

Comment 4

[Peter Radcliffe [:pir]]

 <@nagios-scl3:#sysadmins> pir: fw1.ops.ber2.mozilla.net:Juniper 
  temperature is WARNING - WARNING node0.Routing.Engine Temperature is 51C 
  greater than 50C node1.Routing.Engine Temperature is 51C greater than 50C 
  Last Checked: 2016-03-06 05:10:13 PST

Comment 5

[Peter Radcliffe [:pir]]

This page has all the UPS units with temperature probes that are currently monitored: https://nagios.mozilla.org/scl3/cgi-bin/status.cgi?hostgroup=ups-temperature-probe&style=detail

Comment 6

[Peter Radcliffe [:pir]]

Any more details needed here? I'm about to go on PTO for a week...

Comment 7

[Peter Radcliffe [:pir]]

No more details requested. Closing.

Reopen if there is anything else required.