1474449 - t-yosemite-r7-121.test.releng.mdc2.mozilla.com. is unreachable

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Automatic Bug Generator]

Reboot t-yosemite-r7-121.test.releng.mdc2.mozilla.com. 10.51.56.145
Requested by mozilla-auth0/ad|Mozilla-LDAP|dhouse
Relops controller action failed:
2018-07-09T22:15:20.537475 ssh_reboot -l roller -i ssh.key TimeoutExpired
2018-07-09T22:15:20.539436 ipmi ipmi_reset KeyError
2018-07-09T22:15:20.540825 ipmi ipmi_cycle KeyError

Comment 1

[Automatic Bug Generator]

Reboot t-yosemite-r7-121.test.releng.mdc2.mozilla.com. 10.51.56.145
Requested by mozilla-auth0/ad|Mozilla-LDAP|dhouse
Relops controller action failed:
2018-07-09T22:48:24.655618 ssh_reboot -l roller -i ssh.key TimeoutExpired
2018-07-09T22:53:34.636777 ipmi ipmi_reset KeyError
2018-07-09T22:53:34.640058 ipmi ipmi_cycle KeyError
2018-07-09T22:53:34.640438 ipmi_on  NotImplementedError

Comment 2

[:dhouse]

Please physically netboot/reimage this machines.

I was able to manually reboot it through snmp But it does not come up to respond to ping:
```
[[e-mail] rollerprod]# snmpset -v 2c -c secret pdu1.gc133.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.2.1.9 i 2
iso.3.6.1.4.1.1718.3.2.3.1.11.2.1.9 = INTEGER: 2
[[e-mail] rollerprod]# snmpget -v 2c -c public pdu1.gc133.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.1.9
iso.3.6.1.4.1.1718.3.2.3.1.5.2.1.9 = INTEGER: 0
[[e-mail] rollerprod]# snmpset -v 2c -c secret pdu1.gc133.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.2.1.9 i 1
iso.3.6.1.4.1.1718.3.2.3.1.11.2.1.9 = INTEGER: 1
[[e-mail] rollerprod]# snmpget -v 2c -c public pdu1.gc133.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.1.9
iso.3.6.1.4.1.1718.3.2.3.1.5.2.1.9 = INTEGER: 1
[[e-mail] rollerprod]# ping t-yosemite-r7-121.test.releng.mdc2.mozilla.com
PING t-yosemite-r7-121.test.releng.mdc2.mozilla.com (10.51.56.145) 56(84) bytes of data.
^C
--- t-yosemite-r7-121.test.releng.mdc2.mozilla.com ping statistics ---
62 packets transmitted, 0 received, 100% packet loss, time 61486ms

```

The last successful deploystudio imaging log for this machine is May 24th. Right after that we have a puppet certificate and first run log.
June 2nd we had some failed puppet runs (network connection).
It looks like it broke between then and last week.

Comment 3

[Van Le [:van]]

opened REQ0236225 with QTS to reimage mini.

Comment 4

[:dhouse]

Deploystudio logs came through for the reimage.

Thankyou!