1480258 - t-yosemite-r7-146.test.releng.mdc2.mozilla.com. is unreachable

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Automatic Bug Generator]

Reboot t-yosemite-r7-146.test.releng.mdc2.mozilla.com. 10.51.56.169
Requested by mozilla-auth0/ad|Mozilla-LDAP|dhouse
Relops controller action failed:
2018-08-01T22:55:41.123016 ssh_reboot -l roller -i ssh.key CommandError
2018-08-01T22:56:41.843579 ipmi ipmi_reset KeyError
2018-08-01T22:56:41.847211 ipmi ipmi_cycle KeyError

Comment 1

[Automatic Bug Generator]

Reboot t-yosemite-r7-146.test.releng.mdc2.mozilla.com. 10.51.56.169
Requested by mozilla-auth0/ad|Mozilla-LDAP|dhouse
Relops controller action failed:
2018-08-01T23:00:34.932419 ssh_reboot -l roller -i ssh.key CommandError
2018-08-01T23:01:35.556214 ipmi ipmi_reset KeyError
2018-08-01T23:01:35.561123 ipmi ipmi_cycle KeyError

Comment 2

[:dhouse]

Responds to ping but ssh is refused. I cycled it manually with snmp, and tried leaving it powered off for 30s before powering back on. No change.
```
# snmpset -v 2c -c secret pdu1.gc132.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.14 i 2
iso.3.6.1.4.1.1718.3.2.3.1.11.1.1.14 = INTEGER: 2
# sleep 30; snmpset -v 2c -c secret pdu1.gc132.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.14 i 1
iso.3.6.1.4.1.1718.3.2.3.1.11.1.1.14 = INTEGER: 1
# ping t-yosemite-r7-146.test.releng.mdc2.mozilla.com                      PING t-yosemite-r7-146.test.releng.mdc2.mozilla.com (10.51.56.169) 56(84) bytes of data.
64 bytes from t-yosemite-r7-146.test.releng.mdc2.mozilla.com (10.51.56.169): icmp_seq=1 ttl=62 time=81.5 ms
64 bytes from t-yosemite-r7-146.test.releng.mdc2.mozilla.com (10.51.56.169): icmp_seq=2 ttl=62 time=81.6 ms
^C
--- t-yosemite-r7-146.test.releng.mdc2.mozilla.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 81.526/81.594/81.663/0.293 ms
# # ssh
OpenSSH_7.4p1 Debian-10+deb9u3, OpenSSL 1.0.2l  25 May 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "t-yosemite-r7-146.test.releng.mdc2.mozilla.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to t-yosemite-r7-146.test.releng.mdc2.mozilla.com [10.51.56.169] port 22.
debug1: connect to address 10.51.56.169 port 22: Connection refused
ssh: connect to host t-yosemite-r7-146.test.releng.mdc2.mozilla.com port 22: Connection refused
```

Comment 3

[Van Le [:van]]

opened REQ0240254 with QTS for reimage

Comment 4

[Van Le [:van]]

vle@DESKTOP-3HK51T3:~$ fping t-yosemite-r7-146.test.releng.mdc2.mozilla.com
t-yosemite-r7-146.test.releng.mdc2.mozilla.com is alive