1472841 - (t-yosemite-r7-260) [MDC1] t-yosemite-r7-260 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Comment 1

[:dhouse]

I've confirmed, no response to ping or ssh this morning.

I tried rebooting (with roller) and it did not come back.
Listed in tc, https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc1/t-yosemite-r7-260

no logs found:
(OSX Talos) https://papertrailapp.com/groups/1223184/events?q=t-yosemite-r7-260&focus=957969798439002156
(All Systems) https://papertrailapp.com/groups/1141234/events?q=t-yosemite-r7-260&focus=957969798439002156

Comment 2

[:dhouse]

rebooted because it was refusing ssh
after reboot, I'm able to ssh in and the worker is running
TC shows it started taking tasks:
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc1/t-yosemite-r7-260

```
[[e-mail] ~]$ ssh [e-mail]
The authenticity of host 't-yosemite-r7-260.test.releng.mdc1.mozilla.com (10.49.56.201)' can't be established.
RSA key fingerprint is dd:f4:76:42:fd:91:e2:5d:65:f5:30:e5:47:5c:e1:e0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 't-yosemite-r7-260.test.releng.mdc1.mozilla.com,10.49.56.201' (RSA) to the list of known hosts.
This host is set to follow security level "low"
Unauthorized access prohibited
[[e-mail] ~]# ps -ef|grep worker
   28  1531     1   0  2:26PM ??         0:00.00 /bin/bash /usr/local/bin/run-generic-worker.sh run --config /etc/generic-worker.config
   28  1533  1531   0  2:26PM ??         0:00.37 /usr/local/bin/generic-worker run --config /etc/generic-worker.config
   28  1534  1531   0  2:26PM ??         0:00.02 logger -t generic-worker -s
    0  1549  1539   0  2:26PM ttys000    0:00.00 grep worker
[[e-mail] ~]# uptime
14:26  up 2 mins, 2 users, load averages: 3.85 1.84 0.75
[[e-mail] ~]# ls -ltr /Users/cltbld/tasks/task*/logs/
total 40
-rw-r--r--  1 cltbld  staff     0 Jul 27 14:26 log_warning.log
-rw-r--r--  1 cltbld  staff     0 Jul 27 14:26 log_fatal.log
-rw-r--r--  1 cltbld  staff     0 Jul 27 14:26 log_error.log
-rw-r--r--  1 cltbld  staff     0 Jul 27 14:26 log_critical.log
-rw-r--r--  1 cltbld  staff  1906 Jul 27 14:26 localconfig.json
-rw-r--r--  1 cltbld  staff  4314 Jul 27 14:26 log_raw.log
-rw-r--r--  1 cltbld  staff  5574 Jul 27 14:26 log_info.log
```

Comment 3

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

The machine was having problems again, manage to reboot it and recovered.

Last log was "/usr/local/bin/proxy_reset_env.sh && PUPPET_SERVER=releng-puppet2.srv.releng.mdc1.mozilla.com /usr/local/bin/renew_cert.sh > /dev/null 2>&1"

Took job immediately after reboot.
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc1/t-yosemite-r7-260

Comment 4

[Zsolt Fay [:zfay]]

Still running alive and well.
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc1/t-yosemite-r7-260