1320737 - Generic Worker 7.1.1 crash on win7

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Pete Moore [:pmoore][:pete]]

From https://papertrailapp.com/systems/557666083/events:

Nov 28 16:59:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 2016/11/28 15:59:18 Making system call ResumeThread with args: [540] 
Nov 28 16:59:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 2016/11/28 15:59:18   Result: 1 
Nov 28 17:00:05 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com HaltOnIdle: Is-Running :: generic-worker is running. 
Nov 28 17:00:05 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com HaltOnIdle: instance appears to be productive. 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 2016/11/28 16:01:19 Making system call GetProcessMemoryInfo with args: [558 1216E000 2C] 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 2016/11/28 16:01:19   Result: 1 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 2016/11/28 16:01:19 Uploading artifact: public/logs/localconfig.json 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: unexpected fault address 0x128f8584 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: fatal error: fault 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: [signal 0xc0000005 code=0x1 addr=0x128f8584 pc=0x6b0073] 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: goroutine 1 [running]: 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: runtime.throw(0x8ddf73, 0x5) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/runtime/panic.go:566 +0x7f fp=0x1210f390 sp=0x1210f384 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: runtime.sigpanic() 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/runtime/signal_windows.go:164 +0x116 fp=0x1210f3a8 sp=0x1210f390 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: golang.org/x/crypto/openpgp/packet.parseSignatureSubpacket(0x8aff20, 0x121de3a0, 0x1210f4a8, 0x1210f4b0, 0x120f60d8, 0x121dd470, 0x27, 0x121dd4a0, 0x2b, 0x939ef0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/gopath/src/golang.org/x/crypto/openpgp/packet/signature.go:258 +0x393 fp=0x1210f494 sp=0x1210f3a8 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net/textproto.MIMEHeader.Set(0x0, 0x0, 0x0, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/textproto/header.go:22 +0xa4 fp=0x1210f4c0 sp=0x1210f494 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: goroutine 51 [select, 2 minutes]: 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: main.SignedURLsManager.func2(0x12147f00, 0x1214b03c, 0x122411c0, 0x121541b0, 0x121541c0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/gopath/src/github.com/taskcluster/generic-worker/signedurlmanager.go:57 +0x10a 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: created by main.SignedURLsManager 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/gopath/src/github.com/taskcluster/generic-worker/signedurlmanager.go:73 +0x1e0 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: goroutine 167 [sleep, 24 minutes]: 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: time.Sleep(0xec658540, 0x4e8) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/runtime/time.go:59 +0xd4 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: main.(*TaskRun).setMaxRunTimer.func1(0x12276800) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:974 +0x85 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: created by main.(*TaskRun).setMaxRunTimer 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:977 +0x32 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: goroutine 250 [select, 2 minutes]: 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net/http.(*persistConn).roundTrip(0x11fc54d0, 0x11f7bc30, 0x0, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/http/transport.go:1840 +0xba1 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net/http.(*Transport).RoundTrip(0x11ff20a0, 0x1226a300, 0x11ff20a0, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/http/transport.go:380 +0xc82 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net/http.send(0x1226a300, 0xac77d0, 0x11ff20a0, 0x0, 0x0, 0x0, 0x0, 0x11f7bc20, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/http/client.go:256 +0x51b 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net/http.(*Client).send(0x1242afc0, 0x1226a300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/http/client.go:146 +0x127 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net/http.(*Client).doFollowingRedirects(0x1242afc0, 0x1226a300, 0x93a588, 0x3, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/http/client.go:528 +0x95f 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net/http.(*Client).Do(0x1242afc0, 0x1226a300, 0x11ff20a0, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/http/client.go:187 +0x16a 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: github.com/taskcluster/generic-worker/livelog.(*LiveLog).connectInputStream.func1(0x1242afc0, 0x1226a300) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/gopath/src/github.com/taskcluster/generic-worker/livelog/livelog.go:111 +0x35 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: created by github.com/taskcluster/generic-worker/livelog.(*LiveLog).connectInputStream 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/gopath/src/github.com/taskcluster/generic-worker/livelog/livelog.go:112 +0x221 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: goroutine 273 [IO wait, 2 minutes]: 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net.runtime_pollWait(0x1c18ee8, 0x72, 0x1216c150) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/runtime/netpoll.go:160 +0x55 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net.(*pollDesc).wait(0x12194a70, 0x72, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/fd_poll_runtime.go:73 +0x35 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net.(*ioSrv).ExecIO(0x120d41a0, 0x121949d8, 0x8dec5e, 0x7, 0x93a448, 0x0, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/fd_windows.go:171 +0x122 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net.(*netFD).Read(0x121949a0, 0x121e0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/fd_windows.go:426 +0x129 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net.(*conn).Read(0x11f92000, 0x121e0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/net.go:173 +0xa9 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net/http.(*persistConn).Read(0x11fc54d0, 0x121e0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/http/transport.go:1261 +0x180 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: bufio.(*Reader).fill(0x120fc030) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/bufio/bufio.go:97 +0x172 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: bufio.(*Reader).Peek(0x120fc030, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/bufio/bufio.go:129 +0x8f 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net/http.(*persistConn).readLoop(0x11fc54d0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/http/transport.go:1418 +0x183 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: created by net/http.(*Transport).dialConn 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/http/transport.go:1062 +0xda5 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: goroutine 266 [sleep, 2 minutes]: 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: time.Sleep(0x28f47340, 0x4e9) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/runtime/time.go:59 +0xd4 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: main.(*TaskRun).setMaxRunTimer.func1(0x12276c00) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:974 +0x85 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: created by main.(*TaskRun).setMaxRunTimer 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:977 +0x32 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: goroutine 274 [semacquire]: 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: sync.runtime_notifyListWait(0x1226a2ac, 0x7bc) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/runtime/sema.go:267 +0x17b 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: sync.(*Cond).Wait(0x1226a2a4) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/sync/cond.go:57 +0x8d 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: io.(*pipe).read(0x1226a280, 0x1244c000, 0x8000, 0x8000, 0x0, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/io/pipe.go:47 +0x23a 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: io.(*PipeReader).Read(0x12280028, 0x1244c000, 0x8000, 0x8000, 0x86b140, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/io/pipe.go:129 +0x47 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: io.(*multiReader).Read(0x11f7bdb0, 0x1244c000, 0x8000, 0x8000, 0x3b, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/io/multi.go:20 +0xfc 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: io.copyBuffer(0x32850000, 0x122803b0, 0xac75f0, 0x11f7bdb0, 0x1244c000, 0x8000, 0x8000, 0x243bc, 0x0, 0x0, ...) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/io/io.go:390 +0x1ce 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: io.Copy(0x32850000, 0x122803b0, 0xac75f0, 0x11f7bdb0, 0x122803b0, 0x8dcf47, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/io/io.go:360 +0x53 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net/http.(*transferWriter).WriteBody(0x11face00, 0xac7930, 0x122803a8, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/http/transfer.go:220 +0x22a 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net/http.(*Request).write(0x1226a300, 0xac6cb0, 0x120d6240, 0x0, 0x120d6260, 0x0, 0x0, 0x0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/http/request.go:565 +0x9ac 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: net/http.(*persistConn).writeLoop(0x11fc54d0) 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/http/transport.go:1649 +0x147 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: created by net/http.(*Transport).dialConn 
Nov 28 17:01:21 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: 	/home/travis/go/src/net/http/transport.go:1063 +0xdc5 

This was task run https://tools.taskcluster.net/task-inspector/#ZZls-aAwR5acUwwmn4FVnA/0

Config was:

Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: { 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "accessToken": "*******************************************", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "cachesDir": "Z:\\caches", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "certificate": "{\"version\":1,\"scopes\":[\"assume:worker-type:aws-provisioner-v1/gecko-t-win7-32\",\"assume:worker-id:*\"],\"start\":1480347117773,\"expiry\":1480693017773,\"seed\":\"*****************************************\",\"signature\":\"********************************************\"}", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "checkForNewDeploymentEverySecs": 1800, 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "cleanUpTaskDirs": true, 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "clientId": "project/taskcluster/aws-provisioner/production", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "deploymentId": "4ffb3e0a1af2", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "downloadsDir": "Z:\\downloads", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "idleShutdownTimeoutSecs": 3600, 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "instanceId": "i-0787191c92b4da411", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "instanceType": "c4.2xlarge", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "livelogCertificate": "C:\\generic-worker\\livelog.crt", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "livelogExecutable": "C:\\generic-worker\\livelog.exe", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "livelogKey": "C:\\generic-worker\\livelog.key", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "livelogSecret": "********************************************", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "numberOfTasksToRun": 0, 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "privateIP": "172.31.20.155", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "provisionerId": "aws-provisioner-v1", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "publicIP": "54.215.217.3", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "refreshURLsPrematurelySecs": 310, 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "region": "us-west-1c", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "requiredDiskSpaceMegabytes": 10240, 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "runTasksAsCurrentUser": false, 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "shutdownMachineOnInternalError": true, 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "signingKeyLocation": "not-configured", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "subdomain": "taskcluster-worker.net", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "tasksDir": "Z:\\", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "workerGroup": "us-west-1", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "workerId": "i-0787191c92b4da411", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "workerType": "gecko-t-win7-32", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   "workerTypeMetadata": { 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:     "aws": { 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:       "ami-id": "ami-7d81d51d", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:       "availability-zone": "us-west-1c", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:       "instance-id": "i-0787191c92b4da411", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:       "instance-type": "c4.2xlarge", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:       "local-ipv4": "172.31.20.155", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:       "public-hostname": "ec2-54-215-217-3.us-west-1.compute.amazonaws.com", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:       "public-ipv4": "54.215.217.3" 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:     }, 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:     "generic-worker": { 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:       "go-arch": "386", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:       "go-os": "windows", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:       "go-version": "go1.7.3", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:       "release": "https://github.com/taskcluster/generic-worker/releases/tag/v7.1.1", 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:       "version": "7.1.1" 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:     }, 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:     "machine-setup": { 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:       "manifest": "https://github.com/mozilla-releng/OpenCloudConfig/blob/4ffb3e0a1af22accae6f42e42ba6bffcdda985fe/userdata/Manifest/gecko-t-win7-32.json" 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:     } 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker:   } 
Nov 28 16:37:08 i-0787191c92b4da411.gecko-t-win7-32.usw1.mozilla.com generic-worker: }

Comment 1

[Pete Moore [:pmoore][:pete]]

This looks to me like go routine 1 is writing to a map, presumably while it is being read by a different go routine:

https://github.com/golang/go/blob/go1.7.3/src/net/textproto/header.go#L22

This is an internal (unexported) map of the net/textproto standard library, so I think this could possibly be a go 1.7.3 bug.

Comment 2

[Pete Moore [:pmoore][:pete]]

Another possibility is that this was a consequence of memory corruption due to bug 1320313, subsequently fixed in commit https://github.com/taskcluster/runlib/commit/e882980860e5b33e6cdc9aeaf944da3c310ebbad but not yet rolled out to production on Windows 7 (currently being rolled out to Windows 10 in https://github.com/mozilla-releng/OpenCloudConfig/pull/22).

If the new release plays nicely on Windows 10, we'll roll out to Windows 7 (and eventually Windows Server 2012 R2 too) and then we'll see if we still get this crash stack or not.