851522 - lost IPMI connections during power outage

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Amy Rich [:arr] [:arich]]

We lost the ipmi connections to the ix machines during the power outage.  I was able to get back the ones on the linux machines with ipmitoo, but the windows machines will need to be manually reset.

t-w864-ix-098-mgmt.inband.releng.scl3.mozilla.com
t-w864-ix-099-mgmt.inband.releng.scl3.mozilla.com
t-w864-ix-100-mgmt.inband.releng.scl3.mozilla.com

Comment 1

[Q]

These too:
ix-mn-w732-001-mgmt.inband.releng.scl3.mozilla.com
ix-mn-w732-002-mgmt.inband.releng.scl3.mozilla.com
ix-mn-w732-003-mgmt.inband.releng.scl3.mozilla.com

ix-mn-xp-003-mgmt.inband.releng.scl3.mozilla.com

Comment 2

[Vinh Hua [:vinh]]

All hosts have been rebooted and IPMI verified working.

ix-mn-w732-001-mgmt.inband.releng.scl3.mozilla.com address not found
t-w864-ix-098-mgmt.inband.releng.scl3.mozilla.com is alive
t-w864-ix-099-mgmt.inband.releng.scl3.mozilla.com is alive
t-w864-ix-100-mgmt.inband.releng.scl3.mozilla.com is alive
ix-mn-w732-002-mgmt.inband.releng.scl3.mozilla.com is alive
ix-mn-w732-003-mgmt.inband.releng.scl3.mozilla.com is alive
ix-mn-xp-003-mgmt.inband.releng.scl3.mozilla.com is alive


"ix-mn-w732-001-mgmt.inband.releng.scl3.mozilla.com address not found"  <---- wrong DNS name.  But it's pingable by IP address of 10.26.16.11

$ host 10.26.16.11
11.16.26.10.in-addr.arpa domain name pointer ix-mn-w832-001-mgmt.inband.releng.scl3.mozilla.com.

Comment 3

[Vinh Hua [:vinh]]

DNS updated to match:

vhua$ host 10.26.16.11
11.16.26.10.in-addr.arpa domain name pointer ix-mn-w732-001-mgmt.inband.releng.scl3.mozilla.com.