Closed Bug 1308703 Opened 9 years ago Closed 9 years ago

generic worker 6.0.1 panic on win7

Categories

(Taskcluster :: Workers, defect)

Product:
Taskcluster
Component:
Workers
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: grenade, Unassigned)

Details

latest g-w patches in 6.0.1 seem to work well on win10. on win7 which runs the 32 bit g-w client, instances create the following logs just before they are terminated by HaltOnIdle. Oct 08 09:40:04 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com HaltOnIdle: Is-Running :: generic-worker is running. Oct 08 09:40:04 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com HaltOnIdle: instance appears to be productive. Oct 08 09:40:05 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: 2016/10/08 08:40:04 Reclaimed task HTfj-gBYSx20VA5ndxMboQ successfully. Oct 08 09:42:04 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com HaltOnIdle: drive z: exists Oct 08 09:42:04 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com HaltOnIdle: Is-Running :: generic-worker is running. Oct 08 09:42:04 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com HaltOnIdle: instance appears to be productive. Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: panic: runtime error: invalid memory address or nil pointer dereference Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: [signal 0xc0000005 code=0x1 addr=0x0 pc=0x68914a] Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: goroutine 97 [running]: Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: panic(0x79eb00, 0x11d02030) Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: /home/travis/go/src/runtime/panic.go:500 +0x331 Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: github.com/taskcluster/generic-worker/os.(*Process).signal(0x0, 0x96d740, 0x11cea0bc, 0x0, 0x0) Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/os/exec_windows.go:147 +0x19a Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: github.com/taskcluster/generic-worker/os.(*Process).Signal(0x0, 0x96d740, 0x11cea0bc, 0x0, 0x0) Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/os/doc_windows.go:57 +0x3b Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: github.com/taskcluster/generic-worker/os.(*Process).kill(0x0, 0x0, 0x0) Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/os/exec_windows.go:176 +0x3f Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: github.com/taskcluster/generic-worker/os.(*Process).Kill(0x0, 0x0, 0x0) Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/os/doc_windows.go:51 +0x2b Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: main.(*TaskRun).abortProcess(0x11fb2400, 0x12038440) Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/plat_windows.go:625 +0x7f Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: main.(*TaskRun).kill(0x11fb2400) Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:986 +0x54 Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: main.(*TaskRun).setMaxRunTimer.func1(0x11fb2400) Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:980 +0x68 Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: created by main.(*TaskRun).setMaxRunTimer Oct 08 09:43:29 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:981 +0x32 Oct 08 09:43:41 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com Microsoft-Windows-DSC: The DscTimer is running LCM method PerformRequiredConfigurationChecks with the flag set to 1. Oct 08 09:43:41 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com Microsoft-Windows-DSC: The local configuration manager started. Oct 08 09:43:41 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com Microsoft-Windows-DSC: Job {4E349C11-8D33-11E6-A349-027E07BD7372} : Operation Consistency Check or Pull started by user sid S-1-5-20 from computer NULL. Oct 08 09:43:41 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com Microsoft-Windows-DSC: The DscTimer has successfully run LCM method PerformRequiredConfigurationChecks with flag 1. Oct 08 09:43:41 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com Microsoft-Windows-DSC: The DscTimer is running LCM method PerformRequiredConfigurationChecks with the flag set to 5. Oct 08 09:43:41 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com Microsoft-Windows-DSC: Job {4E349C12-8D33-11E6-A349-027E07BD7372} : Operation Consistency Check or Pull started by user sid S-1-5-20 from computer NULL. Oct 08 09:43:41 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com Microsoft-Windows-DSC: The DscTimer has successfully run LCM method PerformRequiredConfigurationChecks with flag 5. Oct 08 09:44:04 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com HaltOnIdle: drive z: exists Oct 08 09:44:04 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com HaltOnIdle: Is-Running :: generic-worker is not running. Oct 08 09:44:04 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com HaltOnIdle: Is-Running :: OpenCloudConfig is not running. Oct 08 09:44:04 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com HaltOnIdle: Get-Uptime :: last boot: 10/8/2016 7:42:47 AM; uptime: 01:01:15.8010000. Oct 08 09:44:04 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com HaltOnIdle: instance failed validity check and will be halted. Oct 08 09:44:04 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com USER32: The process C:\windows\system32\shutdown.exe (I-004013FFC0A7A) has initiated the shutdown of computer I-004013FFC0A7A on behalf of user NT AUTHORITY\SYSTEM for the following reason: Application: Maintenance (Planned) Reason Code: 0x80040001 Shutdown Type: shutdown Comment: HaltOnIdle :: instance failed validity checks Oct 08 09:44:06 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com Microsoft-Windows-User_Profiles_Service: <12>1 2016-10-08T08:44:03.677800+00:00 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com Microsoft-Windows-User_Profiles_Service 1048 - [NXLOG@14506 Keywords="-9223372036854775808" EventType="WARNING" EventID="1530" ProviderGuid="{89B1E9F0-5AFF-44A6-9B44-0A07A7CE5845}" Version="0" Task="0" OpcodeValue="0" RecordNumber="31603" ThreadID="3432" Channel="Application" Domain="NT AUTHORITY" AccountName="SYSTEM" UserID="SYSTEM" AccountType="User" Opcode="Info" Detail="2 user registry handles leaked from \\Registry\\User\\S-1-5-21-2192203079-3471170106-2880594189-1012: Oct 08 09:44:06 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com Microsoft-Windows-User_Profiles_Service: Process 1076 (\\Device\\HarddiskVolume1\\Windows\\System32\\conhost.exe) has opened key \\REGISTRY\\USER\\S-1-5-21-2192203079-3471170106-2880594189-1012 Oct 08 09:44:06 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com Microsoft-Windows-User_Profiles_Service: Process 1076 (\\Device\\HarddiskVolume1\\Windows\\System32\\conhost.exe) has opened key \\REGISTRY\\USER\\S-1-5-21-2192203079-3471170106-2880594189-1012\\Control Panel\\International Oct 08 09:44:06 i-004013ffc0a7ac9e0.gecko-t-win7-32.usw1.mozilla.com Microsoft-Windows-User_Profiles_Service: " EventReceivedTime="2016-10-08 08:44:04" SourceModuleName="eventlog" SourceModuleType="im_msvistalog"] Windows detected your registry file is still in use by other applications or services. The file will be unloaded now. The applications or services that hold your registry file may not function properly afterwards. DETAIL - 2 user registry handles leaked from \Registry\User\S-1-5-21-2192203079-3471170106-2880594189-1012: Process 1076 (\Device\HarddiskVolume1\Windows\System32\conhost.exe) has opened key \REGISTRY\USER\S-1-5-21-2192203079-3471170106-2880594189-1012 Process 1076 (\Device\HarddiskVolume1\Windows\System32\conhost.exe) has opened key \REGISTRY\USER\S-1-5-21-2192203079-3471170106-2880594189-1012\Control Panel\International
looks like win10 too. just found the same error there: Oct 08 11:39:31 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: panic: runtime error: invalid memory address or nil pointer dereference Oct 08 11:39:31 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: [signal 0xc0000005 code=0x0 addr=0x0 pc=0x6ae40d] Oct 08 11:39:31 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: goroutine 92 [running]: Oct 08 11:39:31 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: panic(0x7d5ec0, 0xc04200c0a0) Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: /home/travis/go/src/runtime/panic.go:500 +0x1af Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: github.com/taskcluster/generic-worker/os.(*Process).signal(0x0, 0xa12920, 0xc04203c200, 0xc04202a370, 0x100000000) Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/os/exec_windows.go:147 +0x2d Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: github.com/taskcluster/generic-worker/os.(*Process).Signal(0x0, 0xa12920, 0xc04203c200, 0xc042365f28, 0x47e4a0) Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/os/doc_windows.go:57 +0x46 Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: github.com/taskcluster/generic-worker/os.(*Process).kill(0x0, 0xc042365f18, 0xc042365f28) Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/os/exec_windows.go:176 +0x4a Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: github.com/taskcluster/generic-worker/os.(*Process).Kill(0x0, 0x8908a8, 0xc04202a370) Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/os/doc_windows.go:51 +0x32 Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: main.(*TaskRun).abortProcess(0xc0420b9800, 0xc04202a370) Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/plat_windows.go:625 +0x90 Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: main.(*TaskRun).kill(0xc0420b9800) Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:986 +0x6e Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: main.(*TaskRun).setMaxRunTimer.func1(0xc0420b9800) Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:980 +0x71 Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: created by main.(*TaskRun).setMaxRunTimer Oct 08 11:39:32 i-0581f1820ad09705a.gecko-t-win10-64-gpu.use1.mozilla.com generic-worker: /home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:981 +0x46
Hey Rob, I think this is fixed in the latest generic worker releases, do you agree? Pete
Flags: needinfo?(rthijssen)
yes
Status: NEW → RESOLVED
Closed: 9 years ago
Flags: needinfo?(rthijssen)
Resolution: --- → FIXED
Component: Generic-Worker → Workers
You need to log in before you can comment on or make changes to this bug.