1473944 - t-yosemite-r7-127.test.releng.mdc2.mozilla.com. is unreachable

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Automatic Bug Generator]

Reboot t-yosemite-r7-127.test.releng.mdc2.mozilla.com. 10.51.56.151
Requested by mozilla-auth0/ad|Mozilla-LDAP|dhouse
Relops controller action failed:
2018-07-06T15:54:51.684349 ssh_reboot -l roller -i ssh.key CommandError
2018-07-06T15:54:51.686595 ipmi ipmi_reset KeyError
2018-07-06T15:54:51.688254 ipmi ipmi_cycle KeyError

Comment 1

[:dhouse]

Please netboot/reimage this machine.

I tried manually with snmp to reboot this machine and that recovered it to respond to ping. But I get a connection refused for ssh (tried from jumphosts in both mdc1 and mdc2 and from roller):
```
# snmpset -v 2c -c secret pdu1.gc133.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.2.1.6 i 2
iso.3.6.1.4.1.1718.3.2.3.1.11.2.1.6 = INTEGER: 2
# snmpget -v 2c -c public pdu1.gc133.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.1.6
iso.3.6.1.4.1.1718.3.2.3.1.5.2.1.6 = INTEGER: 0
# snmpset -v 2c -c secret pdu1.gc133.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.2.1.6 i 1
iso.3.6.1.4.1.1718.3.2.3.1.11.2.1.6 = INTEGER: 1
# snmpget -v 2c -c public pdu1.gc133.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.1.6
iso.3.6.1.4.1.1718.3.2.3.1.5.2.1.6 = INTEGER: 1
[[e-mail] ~]$ ssh -vv [e-mail]
OpenSSH_6.6, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /home/dhouse/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 4: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to t-yosemite-r7-127.test.releng.mdc2.mozilla.com [10.51.56.151] port 22.
debug1: connect to address 10.51.56.151 port 22: Connection refused
ssh: connect to host t-yosemite-r7-127.test.releng.mdc2.mozilla.com port 22: Connection refused
```

Comment 2

[Van Le [:van]]

opened REQ0236225 with QTS to reimage mini.

Comment 3

[Van Le [:van]]

host is back online after reimage.

vle@DESKTOP-3HK51T3:~$ fping !$                                                                                                                
fping t-yosemite-r7-127.test.releng.mdc2.mozilla.com                                                                                           
t-yosemite-r7-127.test.releng.mdc2.mozilla.com is alive