1494083 - t-yosemite-r7-151.test.releng.mdc2.mozilla.com. is unreachable

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Automatic Bug Generator]

Reboot t-yosemite-r7-151.test.releng.mdc2.mozilla.com. 10.51.56.174
Requested by mozilla-auth0/ad|Mozilla-LDAP|riman
Relops controller action failed:
2018-09-25T19:47:06.977558 ssh_reboot -l roller -i ssh.key CommandError
2018-09-25T19:48:07.199898 ipmi ipmi_reset KeyError
2018-09-25T19:48:07.207012 ipmi ipmi_cycle KeyError
2018-09-25T19:48:07.207365 ipmi_on  NotImplementedError

Comment 1

[Van Le [:van]]

back online.

vle@DESKTOP-3HK51T3:~$ fping t-yosemite-r7-151.test.releng.mdc2.mozilla.com
t-yosemite-r7-151.test.releng.mdc2.mozilla.com is alive

Comment 2

[Roland Mutter Michael (:rmutter)]

We cannot ssh into machine once again. Seems online, answer to ping.

Comment 3

[:dhouse]

Can QTS view the screen and check the latest entries in /var/log/system.log if the machine is running? Can we also get screenshots or photos?

Comment 4

[Van Le [:van]]

:dhouse, they can call you when they reach our space and have a crash cart plugged in. can you open a ticket with QTS for this so it's easier to coordinate vs me playing phone tag/irc tag with you? they should be able to take a pic with their phone and send it to you via sms. i believe you have an account on the portal. if you need help or would rather have me do it, let me know.

Comment 5

[:dhouse]

Thanks Van. That sounds exactly like what I need. So I'll ask for them to call me and talk with them for screenshots and to check logs if the machine responds.

Comment 6

[:dhouse]

I created a remote hands request with QTS to check the machine and call me: REQ0249627:RITM0262369:
https://qts.service-now.com/sc_req_item.do?sys_id=2d00b6dbdbf8270c296454e4dc9619e3&sysparm_record_target=sc_req_item&sysparm_catalog=e0d08b13c3330100c8b837659bba8fb4&sysparm_catalog_view=catalog_default

Comment 7

[:dhouse]

QTS found the machine was hung. They rebooted it.