Significant amount of 1064-hw workers Hitting generic-worker Panic
Categories
(Infrastructure & Operations :: RelOps: Windows OS, task)
Tracking
(Not tracked)
People
(Reporter: markco, Assigned: markco)
References
Details
Generic-worker is panicing on on 1064-hw workers:
https://my.papertrailapp.com/groups/1141234/events?focus=1129626380234022915&q=1064-ms%20%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%20PANIC%20occurred%21%20%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A&selected=1129626380234022915
https://my.papertrailapp.com/events?q=1064-ms%20%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%20PANIC%20occurred%21%20%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A .
https://my.papertrailapp.com/groups/1141234/events?focus=1129655992666189835&q=1064-ms-061&selected=1129655992666189835
They seem to have finished task successfully. Then gw panic occurred and the node never recovered. For instances: https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc1/T-W1064-MS-061
Since the significantly high pending I am going to redeploy these nodes. At which time i will be able to get a number of affected nodes.
|
Assignee | |
Comment 2•5 years ago
|
||
The final count is 60 out of 360 deployed hit this issue.
|
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 3•5 years ago
|
||
The immediate work around for this was to remove the json files that could have been causing the issues.
In the wrapper script:
:exit_69
echo "Generic-worker panic! Issue with enviroment or worker bug. (exit code %GW_EXIT_CODE%)" >> <%= $win_generic_worker::generic_worker_dir %>\generic-worker-wrapper.log
echo "Removing possibly corrupted files." >> <%= $win_generic_worker::generic_worker_dir %>\generic-worker-wrapper.log
del /Q /F <%= $win_generic_worker::generic_worker_dir %>\tasks-resolved-count.txt
del /Q /F <%= $win_generic_worker::generic_worker_dir %>\file-caches.json
del /Q /F <%= $win_generic_worker::generic_worker_dir %>\directory-caches.json
del /Q /F <%= $win_generic_worker::generic_worker_dir %>\next-task-user.json
del /Q /F <%= $win_generic_worker::generic_worker_config %> >> <%= $win_generic_worker::generic_worker_dir %>\generic-worker-wrapper.log
This has been addressed in generic-worker generic-worker 16.5.6.
Description
•