1475895 - (t-yosemite-r7-433) [MDC1] t-yosemite-r7-433 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Danut Labici [:dlabici]]

Machine was missing in TC, went ahead with a reimage.
Waiting for the machine to finish it reimage process and come back in TC.

Comment 1

[:dhouse]

This machine is running tasks and reporting logs to papertrail.
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc1/t-yosemite-r7-433
https://papertrailapp.com/groups/1223184?filter=t-yosemite-r7-433

Comment 2

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

Machine is unreachable

bcrisan@bcrisan-P6198:~$ fping t-yosemite-r7-433.test.releng.mdc1.mozilla.com
t-yosemite-r7-433.test.releng.mdc1.mozilla.com is unreachable

Can you guys take a look at it?

Comment 3

[Roland Mutter Michael (:rmutter)]

Hitting that right now: 
channel 2: open failed: connect failed: Connection timed out
Stdio forwarding request failed: Session open refused by peer
ssh_exchange_identification: Connection closed by remote host

Comment 4

[Van Le [:van]]

ive opened REQ0240697  with QTS to reimage the mini.

Comment 5

[Van Le [:van]]

they've performed the task, is the host good now?

Comment 6

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

(In reply to Van Le [:van] from comment #5)
> they've performed the task, is the host good now?

Yes it is, performed more them 20 tasks in the last 10h and almost all of them were successfully completed.

Comment 7

[:dhouse]

According to papertrail, this machine shut down on the 2nd after "Task EpBtY-n6SWCwzodGyHN2Ow finished successfully!" and did not come back up until it was manually rebooted/reimaged by QTS:
```
Aug 02 14:07:06 t-yosemite-r7-433.test.releng.mdc1.mozilla.com generic-worker: 2018/08/02 14:07:06 Resolving task EpBtY-n6SWCwzodGyHN2Ow ...
Aug 02 14:07:06 t-yosemite-r7-433.test.releng.mdc1.mozilla.com generic-worker: 2018/08/02 14:07:06 Task EpBtY-n6SWCwzodGyHN2Ow finished successfully!
Aug 02 14:07:07 t-yosemite-r7-433.test.releng.mdc1.mozilla.com generic-worker: 2018/08/02 14:07:07 Removing task directory '/Users/cltbld/tasks/task_1533243891'...
Aug 02 14:07:12 t-yosemite-r7-433.test.releng.mdc1.mozilla.com generic-worker: 2018/08/02 14:07:12 Resolved 1 tasks in total so far.
Aug 02 14:07:12 t-yosemite-r7-433.test.releng.mdc1.mozilla.com generic-worker: 2018/08/02 14:07:12 Completed all task(s) (number of tasks to run = 1)
Aug 02 14:07:12 t-yosemite-r7-433.test.releng.mdc1.mozilla.com generic-worker: 2018/08/02 14:07:12 Saving file file-caches.json (absolute path: /Users/cltbld/file-caches.json)
Aug 02 14:07:12 t-yosemite-r7-433.test.releng.mdc1.mozilla.com generic-worker: 2018/08/02 14:07:12 Saving file directory-caches.json (absolute path: /Users/cltbld/directory-caches.json)
Aug 02 14:07:12 t-yosemite-r7-433.test.releng.mdc1.mozilla.com generic-worker: 2018/08/02 14:07:12 Exiting worker with exit code 0
Aug 02 14:07:13 t-yosemite-r7-433.test.releng.mdc1.mozilla.com sudo:   cltbld : TTY=unknown ; PWD=/Users/cltbld ; USER=root ; COMMAND=/sbin/reboot
Aug 02 14:07:13 t-yosemite-r7-433.test.releng.mdc1.mozilla.com reboot: rebooted by cltbld
Aug 02 14:07:13 t-yosemite-r7-433.test.releng.mdc1.mozilla.com reboot: SHUTDOWN_TIME: 1533244032 982074
Aug 06 11:14:28 t-yosemite-r7-433.test.releng.mdc1.mozilla.com configd: }
Aug 06 11:14:28 t-yosemite-r7-433.test.releng.mdc1.mozilla.com configd:   1 : State:/Network/Global/IPv4
Aug 06 11:14:28 t-yosemite-r7-433.test.releng.mdc1.mozilla.com DMProxy: AMBD Services: connection interrupted: com.apple.AmbientDisplayAgent (Connection interrupted) 
```

Comment 8

[Radu Iman[:riman]]

This machine is missing from Taskcluster. SSH connection doesn't work.

Logs from Papertrail:

Aug 16 15:26:19 t-yosemite-r7-433 com.apple.xpc.launchd: (com.apple.ReportCrash.Root[33393]): Endpoint has been activated through legacy launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.apple.ReportCrash.DirectoryService
Aug 16 15:26:19 t-yosemite-r7-433 com.apple.xpc.launchd: (com.apple.opendirectoryd[33387]): Service exited due to signal: Illegal instruction: 4
Aug 16 15:26:19 t-yosemite-r7-433 com.apple.xpc.launchd: (com.apple.ReportCrash.Root[33393]): Service exited due to signal: Illegal instruction: 4
Aug 16 15:26:19 t-yosemite-r7-433 com.apple.xpc.launchd: (com.apple.wirelessproxd[33390]): Service exited due to signal: Illegal instruction: 4
Aug 16 15:26:19 t-yosemite-r7-433 com.apple.xpc.launchd: (com.apple.taskgated[33392]): Service exited due to signal: Illegal instruction: 4
Aug 16 15:26:19 t-yosemite-r7-433 com.apple.xpc.launchd: (com.apple.UserEventAgent-System[33388]): Service exited due to signal: Killed: 9
Aug 16 15:26:19 t-yosemite-r7-433 com.apple.xpc.launchd: (com.apple.loginwindow[33389]): Service exited due to signal: Killed: 9
Aug 16 15:26:19 t-yosemite-r7-433 com.apple.xpc.launchd: (com.apple.airportd[33391]): Service exited due to signal: Killed: 9
Aug 16 15:26:19 t-yosemite-r7-433 com.apple.xpc.launchd: (com.apple.ReportCrash.Root): Service only ran for 0 seconds. Pushing respawn out by 10 seconds.

Comment 9

[Adrian Pop]

tried to connect to it today, but the ssh is still unresponsive

Comment 10

[Radu Iman[:riman]]

The machine is running jobs after reboot.
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc1/t-yosemite-r7-433