1320313 - generic worker 7.1.1 crashes on win 10

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Rob Thijssen [:grenade (EET/UTC+0300)]]

The problem is only evident on win 10 (win 7 also uses 7.1.1 and works fine, win 2012 was not updated beyond 7.1.0, so not sure if it would be affected).

I've rolled back win 10 here: https://github.com/mozilla-releng/OpenCloudConfig/commit/4ffb3e0a1af22accae6f42e42ba6bffcdda985fe

below are the last papertrail logs for a win 10 instance which is terminated by HaltOnIdle after the generic worker has crashed:

Nov 25 03:57:50 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com generic-worker:  2016/11/25 11:57:49 Making system call VerSetConditionMask with args: [0 2 3] 
Nov 25 03:57:50 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com generic-worker:  2016/11/25 11:57:49   Result: 8000000000000018 
Nov 25 03:57:50 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com generic-worker:  2016/11/25 11:57:49 Making system call VerSetConditionMask with args: [8000000000000018 1 3] 
Nov 25 03:57:50 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com generic-worker:  2016/11/25 11:57:49   Result: 800000000000001B 
Nov 25 03:57:50 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com generic-worker:  2016/11/25 11:57:49 Making system call VerSetConditionMask with args: [800000000000001B 20 3] 
Nov 25 03:57:50 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com generic-worker:  2016/11/25 11:57:49   Result: 800000000001801B 
Nov 25 03:57:50 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com generic-worker:  2016/11/25 11:57:49 Making system call VerSetConditionMask with args: [800000000001801B 10 3] 
Nov 25 03:57:50 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com generic-worker:  2016/11/25 11:57:49   Result: 800000000001B01B 
Nov 25 03:57:50 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com generic-worker:  2016/11/25 11:57:49 Making system call VerifyVersionInfoW with args: [C0420D8480 33 800000000001B01B] 
Nov 25 03:57:50 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com generic-worker:  2016/11/25 11:57:49   Result: 1 
Nov 25 03:57:50 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com generic-worker:  2016/11/25 11:57:49 Making system call CreateEnvironmentBlock with args: [C0423D0398 4A0 0] 
Nov 25 03:57:50 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com generic-worker:  2016/11/25 11:57:49   Result: 1 
Nov 25 03:57:50 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com generic-worker:  2016/11/25 11:57:49 Making system call DestroyEnvironmentBlock with args: [C0421FE030] 
Nov 25 03:59:01 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com HaltOnIdle:  Is-Running :: generic-worker is not running. 
Nov 25 03:59:02 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com HaltOnIdle:  Is-Running :: OpenCloudConfig is not running. 
Nov 25 03:59:02 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com HaltOnIdle:  Get-Uptime :: last boot: 11/25/2016 11:21:08 AM; uptime: 00:37:52.4378119. 
Nov 25 03:59:02 i-0fad79f21274d1a5f.gecko-t-win10-64.usw1.mozilla.com HaltOnIdle:  instance failed validity check and will be halted.

Comment 1

[Pete Moore [:pmoore][:pete]]

Whhoooooopppps

https://github.com/taskcluster/runlib/commit/e882980860e5b33e6cdc9aeaf944da3c310ebbad

This should fix it!

Comment 2

[Rob Thijssen [:grenade (EET/UTC+0300)]]

rolled out 7.1.2 to win 10 with:
https://github.com/mozilla-releng/OpenCloudConfig/commit/e400c6d2e1ceecb6683739d1e2e7ac7fd9abc3ed

then reverted again to 7.1.0 because of bustage with:
https://github.com/mozilla-releng/OpenCloudConfig/commit/a6c1f1194fe8a662b92fc9be8b49a102eab7f840

Comment 3

[Rob Thijssen [:grenade (EET/UTC+0300)]]

papertrail error log from this mornings new bustage (https://papertrailapp.com/systems/559542273/events):

Nov 29 05:18:02 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com HaltOnIdle:  Is-Running :: generic-worker is running. 
Nov 29 05:18:02 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com HaltOnIdle:  instance appears to be productive. 
Nov 29 05:18:07 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com Microsoft-Windows-DSC:  The local configuration manager was shut down. 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Task found 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Claiming task GCyLDJehRMWTjAgawvy1QQ... 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Deleting task GCyLDJehRMWTjAgawvy1QQ from Azure queue... 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Successfully deleted task from azure queue (delete url: https://taskclusterqueuev1.queue.core.windows.net/queue-b7s5ezzeusysb327h2qdcv4y-ainsukrhble64rgbhljfuvqt-1/messages/ec72333a-2c03-49e1-be49-fd9af5b71aee?popreceipt=AgAAAAMAAAAAAAAAO0yG0ENK0gE%3D&sv=2015-04-05&se=2016-11-29T13%3A44%3A36Z&sp=p&spr=https&sig=SnFEpajDO1ygCZ0OLoAWsf78dovSr8Fo%2Fma8D8HX0wA%3D&st=2016-11-29T12%3A59%3A36Z) with http response code 204. 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Setting reclaim timer... 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Current claim will expire at 2016-11-29 13:38:46.711 +0000 UTC 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Reclaiming 3 mins earlier, at 2016-11-29 13:35:46.711 +0000 UTC 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Time to wait until then is 16m59.8096334s 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 This is more than 30 seconds away - so setting a timer 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 JSON payload: { 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:        "env": { 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          "TMP": "%cd%\\AppData\\Local\\Temp", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          "APPDATA": "%cd%\\AppData\\Roaming", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          "LOCALAPPDATA": "%cd%\\AppData\\Local", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          "TEMP": "%cd%\\AppData\\Local\\Temp" 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:        }, 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:        "artifacts": [ 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          { 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "path": "public\\logs\\localconfig.json", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "expires": "2017-11-29T12:44:48.448567Z", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "type": "file" 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          }, 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          { 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "path": "public\\logs\\log_critical.log", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "expires": "2017-11-29T12:44:48.448567Z", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "type": "file" 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          }, 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          { 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "path": "public\\logs\\log_error.log", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "expires": "2017-11-29T12:44:48.448567Z", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "type": "file" 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          }, 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          { 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "path": "public\\logs\\log_fatal.log", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "expires": "2017-11-29T12:44:48.448567Z", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "type": "file" 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          }, 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          { 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "path": "public\\logs\\log_info.log", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "expires": "2017-11-29T12:44:48.448567Z", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "type": "file" 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          }, 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          { 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "path": "public\\logs\\log_raw.log", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "expires": "2017-11-29T12:44:48.448567Z", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "type": "file" 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          }, 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          { 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "path": "public\\logs\\log_warning.log", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "expires": "2017-11-29T12:44:48.448567Z", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "type": "file" 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          }, 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          { 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "path": "public\\test_info", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "expires": "2017-11-29T12:44:48.448567Z", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:            "type": "directory" 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          } 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:        ], 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:        "maxRunTime": 3600, 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:        "command": [ 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          "mklink /d %cd%\\public\\test_info %cd%\\build\\blobber_upload_dir", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          "mklink %cd%\\public\\logs\\localconfig.json %cd%\\logs\\localconfig.json", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          "mklink %cd%\\public\\logs\\log_critical.log %cd%\\logs\\log_critical.log", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          "mklink %cd%\\public\\logs\\log_error.log %cd%\\logs\\log_error.log", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          "mklink %cd%\\public\\logs\\log_fatal.log %cd%\\logs\\log_fatal.log", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          "mklink %cd%\\public\\logs\\log_info.log %cd%\\logs\\log_info.log", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          "mklink %cd%\\public\\logs\\log_raw.log %cd%\\logs\\log_raw.log", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          "mklink %cd%\\public\\logs\\log_warning.log %cd%\\logs\\log_warning.log", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          "c:\\mozilla-build\\wget\\wget.exe https://queue.taskcluster.net/v1/task/HrpU6Ep9RruMJK_S-uW2eA/artifacts/public/build/mozharness.zip", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          "c:\\mozilla-build\\info-zip\\unzip.exe mozharness.zip", 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:          "c:\\mozilla-build\\python\\python.exe -u mozharness\\scripts\\desktop_unittest.py --cfg mozharness\\configs\\unittests\\win_taskcluster_unittest.py --cppunittest-suite=cppunittest --no-read-buildbot-config --installer-url https://queue.taskcluster.net/v1/task/HrpU6Ep9RruMJK_S-uW2eA/artifacts/public/build/firefox-53.0a1.en-US.win64.zip --test-packages-url https://queue.taskcluster.net/v1/task/HrpU6Ep9RruMJK_S-uW2eA/artifacts/public/build/firefox-53.0a1.en-US.win64.test_packages.json --download-symbols ondemand"  
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:        ], 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:        "osGroups": [], 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:        "mounts": [] 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:      } 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Running task https://tools.taskcluster.net/task-inspector/#GCyLDJehRMWTjAgawvy1QQ/0 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Deadline: 2016-11-29 14:18:46.9033738 +0000 GMT 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Deadline: 2016-11-29 14:18:46.9033738 +0000 GMT 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Deadline: 2016-11-29 14:18:46.9033738 +0000 GMT 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Deadline: 2016-11-29 14:18:46.9033738 +0000 GMT 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Deadline: 2016-11-29 14:18:46.9033738 +0000 GMT 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Deadline: 2016-11-29 14:18:46.9033738 +0000 GMT 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Deadline: 2016-11-29 14:18:46.9033738 +0000 GMT 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Deadline: 2016-11-29 14:18:46.9033738 +0000 GMT 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Deadline: 2016-11-29 14:18:46.9033738 +0000 GMT 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Deadline: 2016-11-29 14:18:46.9033738 +0000 GMT 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Deadline: 2016-11-29 14:18:46.9033738 +0000 GMT 
Nov 29 05:18:47 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:46 Uploading artifact: public/logs/live.log 
Nov 29 05:18:48 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:47 Disk available: 85787103232 bytes 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50 Executing command 0: Z:\task_1480425274\command_000000_wrapper.bat 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50 Script "Z:\\task_1480425274\\command_000000.bat": 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50 Contents: 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50 @echo on 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  mklink /d %cd%\public\test_info %cd%\build\blobber_upload_dir 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  @echo off 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50 Making system call VerSetConditionMask with args: [0 2 3] 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50   Result: 8000000000000018 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50 Making system call VerSetConditionMask with args: [8000000000000018 1 3] 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50   Result: 800000000000001B 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50 Making system call VerSetConditionMask with args: [800000000000001B 20 3] 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50   Result: 800000000001801B 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50 Making system call VerSetConditionMask with args: [800000000001801B 10 3] 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50   Result: 800000000001B01B 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50 Making system call VerifyVersionInfoW with args: [C04240E900 33 800000000001B01B] 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50   Result: 1 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50 Making system call CreateEnvironmentBlock with args: [C0423D1C08 4B0 0] 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50   Result: 1 
Nov 29 05:18:51 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com generic-worker:  2016/11/29 13:18:50 Making system call DestroyEnvironmentBlock with args: [C0423D1C08] 
Nov 29 05:20:02 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com HaltOnIdle:  Is-Running :: generic-worker is not running. 
Nov 29 05:20:02 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com HaltOnIdle:  Is-Running :: OpenCloudConfig is not running. 
Nov 29 05:20:02 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com HaltOnIdle:  Get-Uptime :: last boot: 11/29/2016 1:13:57 PM; uptime: 00:06:04.1516510. 
Nov 29 05:20:02 i-02729dc03939adc8a.gecko-t-win10-64.use1.mozilla.com HaltOnIdle:  instance failed validity check and will be halted.

Comment 4

[Pete Moore [:pmoore][:pete]]

Thanks Rob! And sorry it didn't work. :-(

Looking at these logs, one thing is certainly good - the reference passed to CreateEnvironmentBlock and DestroyEnvironmentBlock is now consistent, e.g.

> Making system call CreateEnvironmentBlock with args: [C0423D1C08 4B0 0] 
> Result: 1 
> Making system call DestroyEnvironmentBlock with args: [C0423D1C08] 

In both cases, 0xC0423D1C08 is passed for the environment block pointer. Previously the values passed were inconsistent.

However, I don't currently have an explanation of why this crashes. The two immediate things I can think of investigating are:

1) On this 64 bit system, are we passing the right number of bits in these sys calls?
2) Is it related to the system call we use the env block in? I don't think so, as we seem to crash before creating the process (but to note that win7 and win10 use different syscalls for creating the process)
3) Could it be related to the interpretation of the environment block, such that on win10 the algorithm to deduce the number of bytes used by each environment variable is not working correctly, due to a slightly different encoding scheme or the presence of env variables that contain special characters not seen on a win 7 environment? Again I don't think it can be this - but worth a look. It depends if we are getting a crash or a panic, which isn't clear from the logs. If there was a panic somewhere before the end of the function, the defer block would get called - so maybe it could be related to the code that reads the environment block. Although that doesn't explain why we see the call, but not the return value...

Comment 5

[Pete Moore [:pmoore][:pete]]

In addition to this, I've requested a win10 VM so I can debug locally, without needing to roll out new workers!

Comment 6

[Rob Thijssen [:grenade (EET/UTC+0300)]]

this was fixed with g-w 7.1.3 which was deployed to all tc windows worker types