1478515 - [MDC2] t-yosemite-r7-130.test.releng.mdc2.mozilla.com. is unreachable

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Automatic Bug Generator]

Reboot t-yosemite-r7-130.test.releng.mdc2.mozilla.com. 10.51.56.154
Requested by mozilla-auth0/ad|Mozilla-LDAP|dhouse
Relops controller action failed:
2018-07-25T23:09:25.869599 ssh_reboot -l roller -i ssh.key TimeoutExpired
2018-07-25T23:14:36.848256 ipmi ipmi_reset KeyError
2018-07-25T23:14:36.851562 ipmi ipmi_cycle KeyError

Comment 1

[:dhouse]

I tried manually powering this machine off and then back on and still no response to ping, and no logs forwarded to papertrail.
```
# snmpget -v 2c -c public pdu1.gc133.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.1.1.4
iso.3.6.1.4.1.1718.3.2.3.1.5.1.1.4 = INTEGER: 1
# snmpset -v 2c -c secret pdu1.gc133.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.4 i 2
iso.3.6.1.4.1.1718.3.2.3.1.11.1.1.4 = INTEGER: 2
# snmpget -v 2c -c public pdu1.gc133.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.1.1.4
iso.3.6.1.4.1.1718.3.2.3.1.5.1.1.4 = INTEGER: 0
# snmpget -v 2c -c public pdu1.gc133.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.1.1.4
iso.3.6.1.4.1.1718.3.2.3.1.5.1.1.4 = INTEGER: 0
# snmpset -v 2c -c secret pdu1.gc133.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.4 i 1
iso.3.6.1.4.1.1718.3.2.3.1.11.1.1.4 = INTEGER: 1
# snmpget -v 2c -c public pdu1.gc133.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.1.1.4
iso.3.6.1.4.1.1718.3.2.3.1.5.1.1.4 = INTEGER: 1
# ping t-yosemite-r7-130.test.releng.mdc2.mozilla.com
PING t-yosemite-r7-130.test.releng.mdc2.mozilla.com (10.51.56.154) 56(84) bytes of data.
^C
--- t-yosemite-r7-130.test.releng.mdc2.mozilla.com ping statistics ---
265 packets transmitted, 0 received, 100% packet loss, time 264473ms
```

Comment 2

[Van Le [:van]]

opened REQ0239167 for reimage.

Comment 3

[Van Le [:van]]

will need to check next visit.

07-26-2018 18:04 EDT - Nicholas Trout Additional comments
Could not pull up display on any of the three effected mac minis. Spoke to Van. He will correct upon his next visit.

Comment 4

[Automatic Bug Generator]

Reboot t-yosemite-r7-130.test.releng.mdc2.mozilla.com. 10.51.56.154
Requested by mozilla-auth0/ad|Mozilla-LDAP|riman
Relops controller action failed:
2018-08-16T22:04:55.281799 ssh_reboot -l roller -i ssh.key TimeoutExpired
2018-08-16T22:10:05.274150 ipmi ipmi_reset KeyError
2018-08-16T22:10:05.276528 ipmi ipmi_cycle KeyError

Comment 5

[Automatic Bug Generator]

Reboot t-yosemite-r7-130.test.releng.mdc2.mozilla.com. 10.51.56.154
Requested by mozilla-auth0/ad|Mozilla-LDAP|riman
Relops controller action failed:
2018-08-21T20:17:24.464583 ssh_reboot -l roller -i ssh.key TimeoutExpired
2018-08-21T20:17:24.469771 ipmi ipmi_reset KeyError
2018-08-21T20:17:24.473172 ipmi ipmi_cycle KeyError

Comment 6

[Zsolt Fay [:zfay]]

Hey Van!
This one was claimed by TC when you last re-imaged it but hasn't taken any jobs since. SSH attempts just hang and is missing from papertrail as well.

Comment 7

[Van Le [:van]]

opened REQ0245333 with QTS for a reimage.

Comment 8

[Van Le [:van]]

back online.

vle@DESKTOP-3HK51T3:~$ fping t-yosemite-r7-130.test.releng.mdc2.mozilla.com
t-yosemite-r7-130.test.releng.mdc2.mozilla.com is alive

Comment 9

[Roland Mutter Michael (:rmutter)]

Seems like the machine is unreachable , last job ended as exception and ping doesn't work. Reopening

Comment 10

[Van Le [:van]]

i'll check this when im on site.

Comment 11

[Van Le [:van]]

i see this host online and running tests.

vle@DESKTOP-3HK51T3:~$ fping  t-yosemite-r7-130.test.releng.mdc2.mozilla.com
t-yosemite-r7-130.test.releng.mdc2.mozilla.com is alive

let me know if you need a reimage.

Comment 12

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

Nope, looks good, no re-image needed.