1472713 - t-yosemite-r7-187.test.releng.mdc2.mozilla.com. is unreachable

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Automatic Bug Generator]

Reboot t-yosemite-r7-187.test.releng.mdc2.mozilla.com. 10.51.56.78
Requested by mozilla-auth0/ad|Mozilla-LDAP|dhouse
Relops controller action failed:
2018-07-02T16:02:40.743539 ssh_reboot -l roller -i ssh.key TimeoutExpired
2018-07-02T16:02:40.745360 ipmi ipmi_reset KeyError
2018-07-02T16:02:40.746705 ipmi ipmi_cycle KeyError
2018-07-02T16:02:46.771093 snmp_reboot pdu1.gc132.ops.releng.mdc2.mozilla.com ba2 CalledProcessError

Comment 1

[Automatic Bug Generator]

Reboot t-yosemite-r7-187.test.releng.mdc2.mozilla.com. 10.51.56.78
Requested by mozilla-auth0/ad|Mozilla-LDAP|dhouse
Relops controller action failed:
2018-07-02T17:26:13.280130 ssh_reboot -l roller -i ssh.key TimeoutExpired
2018-07-02T17:26:13.285198 ipmi ipmi_reset KeyError
2018-07-02T17:26:13.286986 ipmi ipmi_cycle KeyError

Comment 2

[Automatic Bug Generator]

Reboot t-yosemite-r7-187.test.releng.mdc2.mozilla.com. 10.51.56.78
Requested by mozilla-auth0/ad|Mozilla-LDAP|dhouse
Relops controller action failed:
2018-07-02T17:32:23.309790 ssh_reboot -l roller -i ssh.key TimeoutExpired
2018-07-02T17:32:23.314239 ipmi ipmi_reset KeyError
2018-07-02T17:32:23.319508 ipmi ipmi_cycle KeyError
2018-07-02T17:37:18.273395 snmp_reboot pdu1.gc132.ops.releng.mdc2.mozilla.com ba2 SoftTimeLimitExceeded

Comment 3

[Automatic Bug Generator]

Reboot t-yosemite-r7-187.test.releng.mdc2.mozilla.com. 10.51.56.78
Requested by mozilla-auth0/ad|Mozilla-LDAP|dhouse
Relops controller action failed:
2018-07-02T18:43:57.229473 ssh_reboot -l roller -i ssh.key TimeoutExpired
2018-07-02T18:43:57.232143 ipmi ipmi_reset KeyError
2018-07-02T18:43:57.234218 ipmi ipmi_cycle KeyError

Comment 4

[Automatic Bug Generator]

Reboot t-yosemite-r7-187.test.releng.mdc2.mozilla.com. 10.51.56.78
Requested by mozilla-auth0/ad|Mozilla-LDAP|riman
Relops controller action failed:
2018-07-02T21:18:34.323316 ssh_reboot -l roller -i ssh.key TimeoutExpired
2018-07-02T21:18:34.327300 ipmi ipmi_reset KeyError
2018-07-02T21:18:34.329780 ipmi ipmi_cycle KeyError

Comment 5

[Van Le [:van]]

power cycling didn't work, opened REQ0235357  with QTS for remote hands.

Comment 6

[Van Le [:van]]

this one came back online.

vle@DESKTOP-3HK51T3:~$ fping t-yosemite-r7-187.test.releng.mdc2.mozilla.com                                                                      
t-yosemite-r7-187.test.releng.mdc2.mozilla.com is alive

Comment 7

[Zsolt Fay [:zfay]]

Worker is missing from TC. When attempting to ssh into it, it just hangs. I'm re-openig this bug as this worker was busted in the past. let me know if in the future you would like me to contact RelOps first.

Its last logs are from 3 days ago:

Aug 26 23:59:54 t-yosemite-r7-187 kernel: Sandbox: plugin-container(2216) deny iokit-get-properties IOGeneralInterest
Aug 26 23:59:54 t-yosemite-r7-187 kernel: Sandbox: plugin-container(2216) deny iokit-get-properties IOAudioEngineNumActiveUserClients
Aug 27 00:01:36 t-yosemite-r7-187 kernel: process firefox[2214] caught causing excessive wakeups. Observed wakeups rate (per sec): 226; Maximum permitted wakeups rate (per sec): 150; Observation period: 300 seconds; Task lifetime number of wakeups: 45002
Aug 27 00:01:36 t-yosemite-r7-187 com.apple.xpc.launchd: (com.apple.ReportCrash[2232]): Endpoint has been activated through legacy launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.apple.ReportCrash

Comment 8

[Van Le [:van]]

opened REQ0244626  with QTS to reimage. 

in the future if you guys reopen a ticket, can you need info me? otherwise it will get lost in the trail of mac mini bugs that i need to be on site to touch. thanks!

Comment 9

[Zsolt Fay [:zfay]]

We sure can Van!
In the meantime this machine has started taking jobs. It's all good again. Thanks for the help.

https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc2/t-yosemite-r7-187