Closed Bug 1480379 (T-W1064-MS-241) Opened 7 years ago Closed 6 years ago

[MDC1] T-W1064-MS-241 problem tracking

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: arny, Unassigned)

References

Details

Attachments

(1 file)

ms-241.PNG
39.41 KB, image/png
Details
Attached image ms-241.PNG —
Not in TC list and no task for two days, checked ILO and found a failed re-image (see attached capture), reimaged once again and now is taking jobs. PaperTrail logs before reimage: Aug 01 13:48:39 roller1.srv.releng.mdc1.mozilla.com docker-compose: ssl_1 | 10.22.248.9 - - [01/Aug/2018:20:48:37 +0000] "OPTIONS /api/v1/workers/T-W1064-MS-241/jobs?provisioner_id=releng-hardware&worker_type=gecko-t-win10-64-hw&worker_group=mdc1&task_name=reboot HTTP/1.1" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0" "-" Aug 01 13:48:39 roller1.srv.releng.mdc1.mozilla.com docker-compose: web_1 | 172.18.0.1 - - [01/Aug/2018:20:48:37 +0000] "OPTIONS /api/v1/workers/T-W1064-MS-241/jobs HTTP/1.0" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0" Aug 01 13:48:40 roller1.srv.releng.mdc1.mozilla.com docker-compose: ssl_1 | 10.22.248.9 - - [01/Aug/2018:20:48:37 +0000] "POST /api/v1/workers/T-W1064-MS-241/jobs?provisioner_id=releng-hardware&worker_type=gecko-t-win10-64-hw&worker_group=mdc1&task_name=reboot HTTP/1.1" 201 289 "https://tools.taskcluster.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0" "-" Aug 01 13:48:40 roller1.srv.releng.mdc1.mozilla.com docker-compose: web_1 | 172.18.0.1 - - [01/Aug/2018:20:48:37 +0000] "POST /api/v1/workers/T-W1064-MS-241/jobs HTTP/1.0" 201 289 "https://tools.taskcluster.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0" Aug 01 13:51:52 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:51:50,861: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fc8e055f840> (args:('relops_hardware_controller.celery.celery_call_command', '36891f4d-d155-4778-8032-503f0d0abba2', {'lang': 'py', 'task': 'relops_hardware_controller.celery.celery_call_command', 'id': '36891f4d-d155-4778-8032-503f0d0abba2', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '36891f4d-d155-4778-8032-503f0d0abba2', 'parent_id': None, 'argsrepr': "({'http_origin': 'https://tools.taskcluster.net', 'client_id': 'mozilla-auth0/ad|Mozilla-LDAP|riman', 'provisioner_id': 'releng-hardware', 'worker_type': 'gecko-t-win10-64-hw', 'worker_group': 'mdc1', 'worker_id': 't-w1064-ms-241', 'task_name': 'reboot'},)", 'kwargsrepr': '{}', 'origin': 'gen11@41a1de39629b', 'reply_to': '8ed0d9d0-2997-3e55-8232-cb44938d1697', 'correlation_id': '36891f4d-d155-4778-8032-503f0d0abba2', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}}, b'[[{"http_origin": "https://tools.taskcluster.net", "client_id":... kwargs:{}) Aug 01 13:51:52 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:51:50,864: DEBUG/ForkPoolWorker-30] job_data:{'http_origin': 'https://tools.taskcluster.net', 'client_id': 'mozilla-auth0/ad|Mozilla-LDAP|riman', 'provisioner_id': 'releng-hardware', 'worker_type': 'gecko-t-win10-64-hw', 'worker_group': 'mdc1', 'worker_id': 't-w1064-ms-241', 'task_name': 'reboot'} Aug 01 13:51:52 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:51:50,888: INFO/ForkPoolWorker-30] t-w1064-ms-241[10.49.40.152] reboot Aug 01 13:51:52 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:51:50,894: DEBUG/ForkPoolWorker-30] ssh reboot with base args: ssh -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ssh.key -l roller -p 22 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. Aug 01 13:51:57 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:51:55,904: ERROR/ForkPoolWorker-30] Command '['ssh', '-o', 'PasswordAuthentication=no', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-i', 'ssh.key', '-l', 'roller', '-p', '22', 't-w1064-ms-241.wintest.releng.mdc1.mozilla.com.', 'reboot']' timed out after 5 seconds Aug 01 13:51:57 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | subprocess.TimeoutExpired: Command '['ssh', '-o', 'PasswordAuthentication=no', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-i', 'ssh.key', '-l', 'roller', '-p', '22', 't-w1064-ms-241.wintest.releng.mdc1.mozilla.com.', 'reboot']' timed out after 5 seconds Aug 01 13:51:57 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | subprocess.TimeoutExpired: Command '['ssh', '-o', 'PasswordAuthentication=no', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-i', 'ssh.key', '-l', 'roller', '-p', '22', 't-w1064-ms-241.wintest.releng.mdc1.mozilla.com.', 'reboot']' timed out after 5 seconds Aug 01 13:52:14 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:52:12,952: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 1 -w 2 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:52:20 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:52:18,970: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:52:31 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:52:29,990: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:52:41 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:52:40,008: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:52:52 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:52:50,028: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:53:02 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:53:00,047: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:53:12 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:53:10,068: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:53:22 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:53:20,090: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:53:32 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:53:30,102: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:53:42 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:53:40,117: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:53:52 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:53:50,138: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:54:02 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:54:00,151: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:54:12 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:54:10,167: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:54:22 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:54:20,182: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:54:32 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:54:30,198: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:54:42 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:54:40,217: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:54:52 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:54:50,233: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:55:02 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:55:00,250: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:55:12 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:55:10,274: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:55:22 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:55:20,297: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:55:32 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:55:30,318: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:55:42 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:55:40,339: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:55:52 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:55:50,361: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:56:02 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:56:00,379: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:56:12 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:56:10,403: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:56:22 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:56:20,418: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:56:32 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:56:30,439: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:56:42 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:56:40,462: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:56:52 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:56:50,480: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:57:09 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:57:07,897: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 1 -w 2 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:57:15 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:57:13,914: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:57:25 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:57:23,937: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:57:35 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:57:33,954: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:57:46 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:57:43,977: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:57:56 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:57:53,998: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:58:06 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:58:04,019: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:58:16 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:58:14,043: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:58:26 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:58:24,068: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:58:36 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:58:34,086: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:58:46 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:58:44,108: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:58:56 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:58:54,126: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:59:06 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:59:04,147: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:59:16 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:59:14,165: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:59:26 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:59:24,186: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:59:36 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:59:34,202: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:59:46 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:59:44,223: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 13:59:56 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 20:59:54,245: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 14:00:06 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 21:00:04,264: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 14:00:16 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 21:00:14,287: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 14:00:26 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 21:00:24,306: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 14:00:36 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 21:00:34,328: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 14:00:46 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 21:00:44,349: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 14:00:56 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 21:00:54,371: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 14:01:06 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 21:01:04,395: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 14:01:17 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 21:01:15,410: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 14:01:27 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 21:01:25,429: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 14:01:37 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 21:01:35,448: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr ' Aug 01 14:01:47 roller1.srv.releng.mdc1.mozilla.com docker-compose: worker_1 | [2018-08-01 21:01:45,467: WARNING/ForkPoolWorker-30] pinging t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. failed: Command 'set -e;out=$(ping -q -c 4 -w 5 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com. );echo $out|tail -2|tr '
Status: NEW → RESOLVED
Closed: 7 years ago
Depends on: 1452133
Resolution: --- → FIXED
Today was missing again from TC and no jobs for one day. Rebooted and is taking jobs again.
Not was in TC and no jobs for one day. Rebooted but still no jobs. Reimaged and running jobs. Aug 06 02:47:10 T-W1064-MS-241.mdc1.mozilla.com generic-worker: 2018/08/06 09:47:08 Response#015 Aug 06 02:47:10 T-W1064-MS-241.mdc1.mozilla.com generic-worker: 2018/08/06 09:47:08 HTTP/1.1 200 OK#015 Aug 06 02:47:10 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Content-Length: 0#015 Aug 06 02:47:10 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Date: Mon, 06 Aug 2018 09:47:10 GMT#015 Aug 06 02:47:10 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Etag: "d19d85040ced7c705b43767920a69568"#015 Aug 06 02:47:10 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Server: AmazonS3#015 Aug 06 02:47:10 T-W1064-MS-241.mdc1.mozilla.com generic-worker: X-Amz-Id-2: XpD8XNlT3GhmN+q9ZOjIzqz5/FcrCXWzwn/h/LOdyNnmbcjE+B5k8sy2JBuu057w8Hl0LGzfsvg=#015 Aug 06 02:47:10 T-W1064-MS-241.mdc1.mozilla.com generic-worker: X-Amz-Request-Id: C59B78108D1396A7#015 Aug 06 02:47:10 T-W1064-MS-241.mdc1.mozilla.com generic-worker: X-Amz-Version-Id: tiP8lrsqGS90YumZSNnUhO0DMBQwfft7#015 Aug 06 02:47:10 T-W1064-MS-241.mdc1.mozilla.com generic-worker: #015 Aug 06 02:47:10 T-W1064-MS-241.mdc1.mozilla.com generic-worker: 2018/08/06 09:47:08 Redirecting live.log to live_backing.log#015 Aug 06 02:47:10 T-W1064-MS-241.mdc1.mozilla.com generic-worker: 2018/08/06 09:47:08 Uploading artifact: public/logs/live.log#015 Aug 06 02:47:10 T-W1064-MS-241.mdc1.mozilla.com generic-worker: 2018/08/06 09:47:09 Uploading artifact: public/logs/live_backing.log#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: 2018/08/06 09:47:09 Request#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: 2018/08/06 09:47:09 PUT /HM3HCxtRQi2BCLtIjHUvzw/0/public/logs/live_backing.log?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=text%2Fplain%3B%20charset%3Dutf-8&Expires=1533551540&Signature=wso%2BxFXG%2F1H41nUkt7KD%2Bim%2FDXI%3D HTTP/1.1#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Host: taskcluster-public-artifacts.s3-us-west-2.amazonaws.com#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: User-Agent: Go-http-client/1.1#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Content-Length: 20927#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Content-Encoding: gzip#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Content-Type: text/plain; charset=utf-8#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Accept-Encoding: gzip#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: #015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: 2018/08/06 09:47:09 1 put requests issued to https://taskcluster-public-artifacts.s3-us-west-2.amazonaws.com/HM3HCxtRQi2BCLtIjHUvzw/0/public/logs/live_backing.log?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=text%2Fplain%3B%20charset%3Dutf-8&Expires=1533551540&Signature=wso%2BxFXG%2F1H41nUkt7KD%2Bim%2FDXI%3D#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: 2018/08/06 09:47:09 Response#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: 2018/08/06 09:47:09 HTTP/1.1 200 OK#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Content-Length: 0#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Date: Mon, 06 Aug 2018 09:47:11 GMT#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Etag: "5a03f15833c9e52d4f7666118e0ab017"#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Server: AmazonS3#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: X-Amz-Id-2: m5mqovT/O7Ep4jq+hcdjw2jEPLNWrTISJm2AbVxPxmvQGa/0s7mBM02zH0uEP1MDYFaY/s1KzXY=#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: X-Amz-Request-Id: 7E2BE12ECF577B1E#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: X-Amz-Version-Id: MgylzwXtRTj7U_071PeizEOwqw2OvKry#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: #015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: 2018/08/06 09:47:09 Resolving task...#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: 2018/08/06 09:47:09 Command finished successfully!#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: 2018/08/06 09:47:10 No previous task user desktop, so no need to close any open desktops#015 Aug 06 02:47:11 T-W1064-MS-241.mdc1.mozilla.com generic-worker: 2018/08/06 09:47:10 Trying to remove directory 'C:\Users\task_1533545422' via os.RemoveAll(path) call as GenericWorker user...#015 Aug 06 02:47:15 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Graphic Card being used "Intel(R) Iris(R) Pro Graphics P580 " #015 Aug 06 02:47:15 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Removing temp dir contents #015 Aug 06 02:47:15 T-W1064-MS-241.mdc1.mozilla.com generic-worker: C:\Users\GenericWorker\AppData\Local\Temp\aria-debug-7848.log#015 Aug 06 02:47:15 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Deleted file - C:\Users\GenericWorker\AppData\Local\Temp\livelog482020683\stream#015 Aug 06 02:47:15 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Deleted file - C:\ProgramData\Package Cache\{B74E65FD-CC47-41C5-4B89-791A3F61942D}v8.100.25984\Installers\Kits Configuration Installer-x86_en-us.msi#015 Aug 06 02:47:15 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Removing log files older than 1 day #015 Aug 06 02:47:15 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Removing Windows log files older than 7 days #015 Aug 06 02:47:15 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Removing Recycle.bin contents #015 Aug 06 02:47:16 T-W1064-MS-241.mdc1.mozilla.com User32: The process C:\windows\system32\shutdown.exe (T-W1064-MS-241) has initiated the restart of computer T-W1064-MS-241 on behalf of user T-W1064-MS-241\GenericWorker for the following reason: No title for this reason could be found Reason Code: 0x800000ff Shutdown Type: restart Comment: Rebooting as generic worker ran successfully#015 Aug 06 02:47:19 T-W1064-MS-241.mdc1.mozilla.com Service_Control_Manager: The sshd service terminated unexpectedly. It has done this 1 time(s).#015
No jobs for one day, rebooted and now is running tasks. Sep 06 05:54:07 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Checking for manifest completetion #015 Sep 06 05:54:12 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Checking for manifest completetion #015 Sep 06 05:54:19 T-W1064-MS-241.mdc1.mozilla.com Microsoft-Windows-DSC: " Sep 06 05:54:19 t-w1064-ms-241.wintest.releng.mdc1.mozilla.com t-w1064-ms-241.wintest.releng.mdc1.mozilla.com: Completed processing test operation. The operation returned False." EventReceivedTime="2018-09-06 12:54:19" SourceModuleName="eventlog" SourceModuleType="im_msvistalog"] Job {ECB0D285-B1D3-11E8-949B-F40343DF4706} : From LCM, message is Completed processing test operation. The operation returned False.#015 Sep 06 05:54:19 T-W1064-MS-241.mdc1.mozilla.com Microsoft-Windows-DSC: Job {ECB0D285-B1D3-11E8-949B-F40343DF4706} : Consistency engine was run successfully.#015 Sep 06 05:54:19 T-W1064-MS-241.mdc1.mozilla.com Microsoft-Windows-DSC: Job {ECB0D285-B1D3-11E8-949B-F40343DF4706} : Job runs under the following LCM setting. ConfigurationMode: ApplyAndMonitor ConfigurationModeFrequencyMins: 15 RefreshMode: PUSH RefreshFrequencyMins: 30 RebootNodeIfNeeded: NONE DebugMode: False#015 Sep 06 05:54:19 T-W1064-MS-241.mdc1.mozilla.com Microsoft-Windows-DSC: Job {ECB0D285-B1D3-11E8-949B-F40343DF4706} : Operation Consistency Check or Pull completed successfully.#015 Sep 06 05:54:19 T-W1064-MS-241.mdc1.mozilla.com Microsoft-Windows-DSC: Job {ECB0D285-B1D3-11E8-949B-F40343DF4706} : Details logging completed for C:\windows\System32\Configuration\ConfigurationStatus\{ECB0D285-B1D3-11E8-949B-F40343DF4706}-0.details.json.#015 Sep 06 05:54:19 T-W1064-MS-241.mdc1.mozilla.com Microsoft-Windows-DSC: The DscTimer has successfully run LCM method PerformRequiredConfigurationChecks with flag 1.#015 Sep 06 05:54:22 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Checking for manifest completetion #015 Sep 06 05:54:27 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Checking for manifest completetion #015 Sep 06 05:54:37 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Checking for manifest completetion #015 Sep 06 05:54:42 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Checking for manifest completetion #015 Sep 06 05:54:52 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Checking for manifest completetion #015 Sep 06 05:54:58 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Checking for manifest completetion #015 Sep 06 05:55:03 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Checking for manifest completetion #015 Sep 06 05:55:13 T-W1064-MS-241.mdc1.mozilla.com generic-worker: Checking for manifest completetion #015
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Machine is back and running jobs: https://tools.taskcluster.net/groups/CI335uDpQhaV1opEPtCW8A/tasks/XrDtZ_qESheTzPXFidXf2w/runs/0
Status: REOPENED → RESOLVED
Closed: 7 years ago → 6 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: