1485271 - [MDC2] t-yosemite-r7-175.test.releng.mdc2.mozilla.com. is unreachable

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Automatic Bug Generator]

Reboot t-yosemite-r7-175.test.releng.mdc2.mozilla.com. 10.51.56.66
Requested by mozilla-auth0/ad|Mozilla-LDAP|zfay
Relops controller action failed:
2018-08-22T07:06:37.127316 ssh_reboot -l roller -i ssh.key TimeoutExpired
2018-08-22T07:11:47.121903 ipmi ipmi_reset KeyError
2018-08-22T07:11:47.123989 ipmi ipmi_cycle KeyError

Comment 1

[Zsolt Fay [:zfay]]

When attempting to ssh, it times out. These are the last few lines of logs from papertrail: 
Aug 20 19:26:21 t-yosemite-r7-175 kernel: firefox (map: 0xffffff80364e64b0) triggered DYLD shared region unnest for map: 0xffffff80364e64b0, region 0x7fff96e00000->0x7fff97000000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
Aug 20 19:27:59 t-yosemite-r7-175 kernel: Kext loading now disabled.
Aug 20 19:27:59 t-yosemite-r7-175 kernel: Kext unloading now disabled.
Aug 20 19:27:59 t-yosemite-r7-175 kernel: Kext autounloading now disabled.
Aug 20 19:27:59 t-yosemite-r7-175 kernel: Kernel requests now disabled.
Aug 20 19:27:59 t-yosemite-r7-175 com.apple.xpc.launchd: (com.apple.xpc.launchd.domain.system): System reboot initiated by: reboot.2302<-sudo.2301<-bash.1456<-launchd.1 


https://papertrailapp.com/systems/t-yosemite-r7-175/events

Comment 2

[Van Le [:van]]

opened REQ0243439 with QTS for reimage.

Comment 3

[Van Le [:van]]

QTS completed task but unsure what's going on as this mini didn't reimage. will need to poke next site visit.

Comment 4

[Automatic Bug Generator]

Reboot t-yosemite-r7-175.test.releng.mdc2.mozilla.com. 10.51.56.66
Requested by mozilla-auth0/ad|Mozilla-LDAP|dhouse
Relops controller action failed:
2018-08-24T06:14:48.278240 ssh_reboot -l roller -i ssh.key TimeoutExpired
2018-08-24T06:19:58.268059 ipmi ipmi_reset KeyError
2018-08-24T06:19:58.269605 ipmi ipmi_cycle KeyError
2018-08-24T06:19:58.269883 ipmi_on  NotImplementedError

Comment 5

[Automatic Bug Generator]

Reboot t-yosemite-r7-175.test.releng.mdc2.mozilla.com. 10.51.56.66
Requested by mozilla-auth0/ad|Mozilla-LDAP|riman
Relops controller action failed:
2018-09-25T19:54:12.648277 ssh_reboot -l roller -i ssh.key TimeoutExpired
2018-09-25T19:59:22.651746 ipmi ipmi_reset KeyError
2018-09-25T19:59:22.655233 ipmi ipmi_cycle KeyError
2018-09-25T19:59:22.655829 ipmi_on  NotImplementedError

Comment 6

[Automatic Bug Generator]

Reboot t-yosemite-r7-175.test.releng.mdc2.mozilla.com. 10.51.56.66
Requested by mozilla-auth0/ad|Mozilla-LDAP|apop
Relops controller action failed:
2018-10-15T17:03:36.203208 ssh_reboot -l roller -i ssh.key TimeoutExpired
2018-10-15T17:05:38.503905 ipmi ipmi_reset KeyError
2018-10-15T17:05:38.505300 ipmi ipmi_cycle KeyError

Comment 7

[:dhouse]

I tried turning off the power manually and then turning it back on and still get no ping/ssh response

Comment 8

[:dhouse]

I'll file a case with QTS to get it rebooted.

Comment 9

[:dhouse]

Created new case with QTS: REQ0252005
https://qts.service-now.com/sc_request.do?sys_id=db30abc0dbd967cc296454e4dc961906&sysparm_record_target=sc_request&sysparm_catalog=e0d08b13c3330100c8b837659bba8fb4&sysparm_catalog_view=catalog_default

Comment 10

[:dhouse]

QTS called and said that the machine will not turn on. They said when then found it, it was powered off and that it will not power on but is dead completely.

Comment 11

[:dhouse]

Van, QTS found this machine will not power on. Could you put it on the list to replace or check?

Comment 12

[Automatic Bug Generator]

Reboot t-yosemite-r7-175.test.releng.mdc2.mozilla.com. 10.51.56.66
Requested by mozilla-auth0/ad|Mozilla-LDAP|apop
Relops controller action failed:
2018-10-17T18:19:58.556225 ssh_reboot -l roller -i ssh.key TimeoutExpired
2018-10-17T18:22:00.931825 ipmi ipmi_reset KeyError
2018-10-17T18:22:00.933599 ipmi ipmi_cycle KeyError

Comment 13

[Van Le [:van]]

:dhouse, it was noted in c#3 and the whiteboard that this mini needed hands on. sorry you had to go through the troubles of troubleshooting it again.

Comment 14

[:dhouse]

(In reply to Van Le [:van] from comment #13)
> :dhouse, it was noted in c#3 and the whiteboard that this mini needed hands
> on. sorry you had to go through the troubles of troubleshooting it again.

Thanks! No worries. I was thinking it may have been checked already.

Comment 15

[Van Le [:van]]

power issue - back online after reimage.

vle@DESKTOP-3HK51T3:~$ fping t-yosemite-r7-175.test.releng.mdc2.mozilla.com
t-yosemite-r7-175.test.releng.mdc2.mozilla.com is alive