1472874 - monitor c7000s boa and switches in MDC2

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Service Requests, task)

Description

[Van Le [:van]]

please monitor these new hosts

hpswitch1-access.gf135.ops.mdc2.mozilla.net: 10.50.8.17
boa-a1-chassis1.gf135.ops.mdc2.mozilla.com: 10.50.8.32
boa-a2-chassis1.gf135.ops.mdc2.mozilla.com: 10.50.8.33

hpswitch2-access.gf135.ops.mdc2.mozilla.net:10.50.8.18
boa-a1-chassis2.gf135.ops.mdc2.mozilla.com: 10.50.8.50
boa-a2-chassis2.gf135.ops.mdc2.mozilla.com: 10.50.8.51

Comment 1

[Van Le [:van]]

err, these are not hosts but rather onboard administrators and cisco switches.

Comment 2

[Ryan C [:ryanc] (UTC-4)]

(In reply to Van Le [:van] from comment #0)
> please monitor these new hosts
> 
> hpswitch1-access.gf135.ops.mdc2.mozilla.net: 10.50.8.17
> boa-a1-chassis1.gf135.ops.mdc2.mozilla.com: 10.50.8.32
> boa-a2-chassis1.gf135.ops.mdc2.mozilla.com: 10.50.8.33
> 
> hpswitch2-access.gf135.ops.mdc2.mozilla.net:10.50.8.18
> boa-a1-chassis2.gf135.ops.mdc2.mozilla.com: 10.50.8.50
> boa-a2-chassis2.gf135.ops.mdc2.mozilla.com: 10.50.8.51

For the switches, we're only monitoring ping or is there anything else we should be doing with snmp etc?

Comment 3

[Ryan C [:ryanc] (UTC-4)]

Initial stuff pushed in 9c73a0918d31db5765bece7302c2497cdececa22 -- Will report back

Comment 4

[Van Le [:van]]

these should be monitored the same as our other cisco module switches. 

for reference:
switch1.r301-5.ops.scl3.mozilla.net (10.22.8.140)
switch1.r302-1.ops.scl3.mozilla.net (10.22.8.149)

Comment 5

[Ryan C [:ryanc] (UTC-4)]

These are not looking consistent. Please make sure ACLs are in place and that all BOA are using the same community string,

05:17:11 <nagios-mdc2> ryanc: [Unknown] boa-a1-chassis1.gf135.ops.mdc2.mozilla.com:HP Blade Chassis is OK - OK - System: 'BladeSystem c7000 Enclosure G2', SN: 'USE951W478', Firmware: '4.22', hardware working fine, 1 blades, 2 i/o modules Last Checked: 2018-07-04 09:08:25 UTC
05:17:11 <nagios-mdc2> ryanc: [Unknown] boa-a1-chassis2.gf135.ops.mdc2.mozilla.com:HP Blade Chassis is CRITICAL - PSU 6 is Failed (generalFailure), input line status: linePowerLoss<br/>Enclosure overall health condition is Degraded Last Checked: 2018-07-04 09:08:31 UTC
05:17:11 <nagios-mdc2> ryanc: [Unknown] boa-a2-chassis1.gf135.ops.mdc2.mozilla.com:HP Blade Chassis is CRITICAL - SNMP CRITICAL: No response from remote host "10.50.8.33" Last Checked: 2018-07-04 09:08:45 UTC
05:17:11 <nagios-mdc2> ryanc: [Unknown] boa-a2-chassis2.gf135.ops.mdc2.mozilla.com:HP Blade Chassis is CRITICAL - SNMP CRITICAL: No response from remote host "10.50.8.51" Last Checked: 2018-07-04 09:16:50 UTC

Comment 6

[Linear Ni-Ya Li [:nli]]

(IRC) Fri 03:08:32 UTC [9204] [Unknown] boa-a1-chassis2.gf135.ops.mdc2.mozilla.com:HP Agents is CRITICAL: Compaq/HP Agent Check: cpqRackPowerEnclosureCondition (1:degraded) cpqRackPowerSupplyCondition (6:failed) (http://m.mozilla.org/HP+Agents)

(IRC) Fri 03:08:34 UTC [9205] [Unknown] boa-a1-chassis2.gf135.ops.mdc2.mozilla.com:HP Blade Chassis is CRITICAL: PSU 6 is Failed (generalFailure), input line status: linePowerLoss<br/>Enclosure overall health condition is Degraded (http://m.mozilla.org/HP+Blade+Chassis)
 
(IRC) Fri 03:08:57 UTC [9208] [Unknown] boa-a2-chassis1.gf135.ops.mdc2.mozilla.com:HP Blade Chassis is CRITICAL: SNMP CRITICAL: No response from remote host "10.50.8.33" (http://m.mozilla.org/HP+Blade+Chassis)

Downtimed 7d waiting for the fixes for comment5

Comment 7

[Van Le [:van]]

acls are not the issue. it appears since these are on active/standby, only the active boa will respond to snmp. please monitor the "boa-a1*" boards. we can monitor the secondary (a2) for ping since they respond to that.

Comment 8

[Van Le [:van]]

>05:17:11 <nagios-mdc2> ryanc: [Unknown] boa-a1-chassis2.gf135.ops.mdc2.mozilla.com:HP Blade Chassis is CRITICAL - PSU 6 is Failed (generalFailure), input line status: linePowerLoss<br/>Enclosure overall health condition is Degraded Last Checked: 2018-07-04 09:08:31 UTC

just an fyi, i have opened bug 1472871 for the bad PSU.

Comment 9

[Vinh Hua [:vinh]]

Monitoring changes made in commit 59b102d6183fceb02fb55ce5c62a5781c691889b.