1505216 - monitor all UPS for temperature

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Service Requests, task)

Description

[Van Le [:van]]

it doesn't look like TPE1's UPS alerted of us the temperature issue in TPE1 today (11/6/2018) before the devices started shutting down. can we make sure we have all the UPS monitoring temperature or maybe it is but something is broken so it didn't alert in #sysadmins? i see UPS ups-red01.df401-1.private.tpe1 on the observium [1]list though.

van> nagios-mdc1: status ups-red01.df401-1.private.tpe1.mozilla.net:*
3:34 PM 
<nagios-mdc1> van: [networkops] ups-red01.df401-1.private.tpe1.mozilla.net:PING is OK - PING OK - Packet loss = 0%, RTA = 136.26 ms Last Checked: 2018-11-06 23:33:22 UTC
3:34 PM van: [networkops] ups-red01.df401-1.private.tpe1.mozilla.net:UPS Battery Replacement is OK - SNMP OK - Status 1 Last Checked: 2018-11-06 23:32:49 UTC
3:34 PM van: [networkops] ups-red01.df401-1.private.tpe1.mozilla.net:UPS Battery Status is OK - SNMP OK - Status 2 Last Checked: 2018-11-06 23:28:47 UTC
3:34 PM van: [networkops] ups-red01.df401-1.private.tpe1.mozilla.net:UPS Output Status is OK - SNMP OK - Status 2 Last Checked: 2018-11-06 23:31:09 UTC

[1] https://observium1.private.mdc2.mozilla.com/alert_check/alert_test_id=19/

Comment 1

[Van Le [:van]]

i thought observium interacted with the irc bots. is it possible to add the temperature/humidty UPS check to nagios so we get the alerts in #sysadmins? thanks!

Comment 2

[Ryan C [:ryanc] (UTC-4)]

(In reply to Van Le [:van] from comment #0)
> it doesn't look like TPE1's UPS alerted of us the temperature issue in TPE1
> today (11/6/2018) before the devices started shutting down. can we make sure
> we have all the UPS monitoring temperature or maybe it is but something is
> broken so it didn't alert in #sysadmins? i see UPS
> ups-red01.df401-1.private.tpe1 on the observium [1]list though.
> 
> van> nagios-mdc1: status ups-red01.df401-1.private.tpe1.mozilla.net:*
> 3:34 PM 
> <nagios-mdc1> van: [networkops]
> ups-red01.df401-1.private.tpe1.mozilla.net:PING is OK - PING OK - Packet
> loss = 0%, RTA = 136.26 ms Last Checked: 2018-11-06 23:33:22 UTC
> 3:34 PM van: [networkops] ups-red01.df401-1.private.tpe1.mozilla.net:UPS
> Battery Replacement is OK - SNMP OK - Status 1 Last Checked: 2018-11-06
> 23:32:49 UTC
> 3:34 PM van: [networkops] ups-red01.df401-1.private.tpe1.mozilla.net:UPS
> Battery Status is OK - SNMP OK - Status 2 Last Checked: 2018-11-06 23:28:47
> UTC
> 3:34 PM van: [networkops] ups-red01.df401-1.private.tpe1.mozilla.net:UPS
> Output Status is OK - SNMP OK - Status 2 Last Checked: 2018-11-06 23:31:09
> UTC
> 
> [1] https://observium1.private.mdc2.mozilla.com/alert_check/alert_test_id=19/

Yeah it did, https://observium1.private.mdc2.mozilla.com/graphs/to=1541548780/device=25/type=device_temperature/from=1541462380/legend=yes/

https://mozilla.pagerduty.com/incidents/PODKQRR

Comment 3

[Ryan C [:ryanc] (UTC-4)]

(In reply to Van Le [:van] from comment #1)
> i thought observium interacted with the irc bots. is it possible to add the
> temperature/humidty UPS check to nagios so we get the alerts in #sysadmins?
> thanks!

There's an irc bot to query Observium, it doesn't display alerts, but they do go to Slack, which I can invite you to and others to -- Let me know

Comment 4

[Ryan C [:ryanc] (UTC-4)]

And in regards to the title of this bug, "monitor all UPS for temperature", we do monitor all ups', as long as they're in the secrets file we've designated for this host so that Puppet automatically adds them

Comment 5

[Ryan C [:ryanc] (UTC-4)]

(In reply to Ryan C [:ryanc] (UTC-4) from comment #4)
> And in regards to the title of this bug, "monitor all UPS for temperature",
> we do monitor all ups', as long as they're in the secrets file we've
> designated for this host so that Puppet automatically adds them

puppet_secrets/hiera/nodes/observium1.private.mdc2.mozilla.com.yaml

Comment 6

[Van Le [:van]]

thanks for clearing this up :ryanc!