1508195 - [MDC1] t-yosemite-r7-266 is unreachable

Status: RESOLVED INVALID

(Infrastructure & Operations :: RelOps: General, task)

Description

[Roland Mutter Michael (:rmutter)]

It seems like this worker is unreachable. Tried to ssh and ping the machine.

Last papertrail logs:

Nov 16 13:18:25 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1  | [2018-11-16 21:18:23,678: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fa2025369d8> (args:('relops_hardware_controller.celery.celery_call_command', 'd33a019b-8ffd-48f5-bb3e-563449f971bf', {'lang': 'py', 'task': 'relops_hardware_controller.celery.celery_call_command', 'id': 'd33a019b-8ffd-48f5-bb3e-563449f971bf', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd33a019b-8ffd-48f5-bb3e-563449f971bf', 'parent_id': None, 'argsrepr': "({'http_origin': 'https://tools.taskcluster.net', 'client_id': 'mozilla-auth0/ad|Mozilla-LDAP|dhouse', 'provisioner_id': 'releng-hardware', 'worker_type': 'gecko-t-osx-1010', 'worker_group': 'mdc1', 'worker_id': 't-yosemite-r7-266', 'task_name': 'reboot'},)", 'kwargsrepr': '{}', 'origin': 'gen10@db795e5c34d8', 'reply_to': '716e7cef-859c-30d5-b62c-7b8046e8cc9e', 'correlation_id': 'd33a019b-8ffd-48f5-bb3e-563449f971bf', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}}, b'[[{"http_origin": "https://tools.taskcluster.net", "client_id":... kwargs:{})
Nov 16 13:18:25 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1  | [2018-11-16 21:18:23,681: DEBUG/ForkPoolWorker-1] job_data:{'http_origin': 'https://tools.taskcluster.net', 'client_id': 'mozilla-auth0/ad|Mozilla-LDAP|dhouse', 'provisioner_id': 'releng-hardware', 'worker_type': 'gecko-t-osx-1010', 'worker_group': 'mdc1', 'worker_id': 't-yosemite-r7-266', 'task_name': 'reboot'}
Nov 16 13:18:25 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1  | [2018-11-16 21:18:23,694: INFO/ForkPoolWorker-1] MDC1 t-yosemite-r7-266[10.49.56.207] reboot
Nov 16 13:18:25 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1  |  b'{"channel":"#roller","message":"MDC1 t-yosemite-r7-266[10.49.56.207] reboot '
Nov 16 13:18:25 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1  | [2018-11-16 21:18:23,703: DEBUG/ForkPoolWorker-1] HTTP Payload: {"channel":"#roller","message":"MDC1 t-yosemite-r7-266[10.49.56.207] reboot requested by dhouse ..." (limit 100 char)
Nov 16 13:18:25 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1  | [2018-11-16 21:18:24,167: DEBUG/ForkPoolWorker-1] ssh reboot with base args: ssh -o PasswordAuthentication=no -o ServerAliveInterval=2 -o LogLevel=ERROR -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ssh.key -l roller -p 22 t-yosemite-r7-266.test.releng.mdc1.mozilla.com.
Nov 16 13:18:30 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1  | [2018-11-16 21:18:29,182: ERROR/ForkPoolWorker-1] Command '['ssh', '-o', 'PasswordAuthentication=no', '-o', 'ServerAliveInterval=2', '-o', 'LogLevel=ERROR', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-i', 'ssh.key', '-l', 'roller', '-p', '22', 't-yosemite-r7-266.test.releng.mdc1.mozilla.com.', 'reboot']' timed out after 5 seconds
Nov 16 13:18:30 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1  | subprocess.TimeoutExpired: Command '['ssh', '-o', 'PasswordAuthentication=no', '-o', 'ServerAliveInterval=2', '-o', 'LogLevel=ERROR', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-i', 'ssh.key', '-l', 'roller', '-p', '22', 't-yosemite-r7-266.test.releng.mdc1.mozilla.com.', 'reboot']' timed out after 5 seconds
Nov 16 13:18:30 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1  | subprocess.TimeoutExpired: Command '['ssh', '-o', 'PasswordAuthentication=no', '-o', 'ServerAliveInterval=2', '-o', 'LogLevel=ERROR', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-i', 'ssh.key', '-l', 'roller', '-p', '22', 't-yosemite-r7-266.test.releng.mdc1.mozilla.com.', 'reboot']' timed out after 5 seconds
Nov 16 13:18:30 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1  | [2018-11-16 21:18:29,188: INFO/ForkPoolWorker-1] Powercycling t-yosemite-r7-266.test.releng.mdc1.mozilla.com. via pdu1.rit44.ops.releng.mdc1.mozilla.com.
Nov 16 13:18:30 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1  | [2018-11-16 21:18:29,188: INFO/ForkPoolWorker-1] Powering down t-yosemite-r7-266.test.releng.mdc1.mozilla.com. ...
Nov 16 13:19:30 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1  | [2018-11-16 21:19:29,293: INFO/ForkPoolWorker-1] Powering up t-yosemite-r7-266.test.releng.mdc1.mozilla.com. ...
Nov 18 18:59:46 rejh1.srv.releng.mdc1.mozilla.com sshd:  error: connect_to t-yosemite-r7-266.test.releng.mdc1.mozilla.com port 22: failed.

Comment 1

[Zsolt Fay [:zfay]]

Just noticed that Jake reserved this mini in preparation to support macOS 10.14 Mojave. See bug 1487818