1507069 - Generic worker on host t-yosemite-r7-265.test.releng.mdc1.mozilla.com returned exit code 69

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Automatic Bug Generator]

Error code 69 means: worker panic - either a worker bug, or the environment is not suitable for running                a task, e.g. a file cannot be written to the file system, or something else did                not work that was required in order to execute a task. See config setting                shutdownMachineOnInternalError.                The following log were provided:                https://papertrailapp.com/systems/t-yosemite-r7-265.test.releng.mdc1.mozilla.com/events?q=program%3Ageneric-worker

Comment 1

[Roland Mutter Michael (:rmutter)]

This worker seems unreachable.

Last logs from papertrail: 


Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: Attempts: 1
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: 2018/11/16 19:37:03 Querying queue to get latest status for task LUznQhlwSk2yihLFePZ74g...
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: 2018/11/16 19:37:03 Latest status: Errored
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: 2018/11/16 19:37:03 Resolving task LUznQhlwSk2yihLFePZ74g ...
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: 2018/11/16 19:37:03 Not updating status of task LUznQhlwSk2yihLFePZ74g run 0 from Errored to Failed. This is because you can only update to status Failed if the previous status was one of: [Claimed Reclaimed Aborted]
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: 2018/11/16 19:37:03 Saving file file-caches.json (absolute path: /Users/cltbld/file-caches.json)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: 2018/11/16 19:37:03 Saving file directory-caches.json (absolute path: /Users/cltbld/directory-caches.json)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: 2018/11/16 19:37:03 goroutine 1 [running]:
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: runtime/debug.Stack(0x42240ec80, 0x141e80c, 0x1559fee)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/.gimme/versions/go1.10.3.src/src/runtime/debug/stack.go:24 +0xa7
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: main.HandleCrash(0x14ae1e0, 0x421de8ab0)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:562 +0x26
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: main.RunWorker.func1(0x42240fda0)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:581 +0x52
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: panic(0x14ae1e0, 0x421de8ab0)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/.gimme/versions/go1.10.3.src/src/runtime/panic.go:502 +0x229
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: main.(*TaskRun).Run.func1(0x42209c080, 0x4219e8a00)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:1112 +0xc5
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: panic(0x14ae1e0, 0x421de8ab0)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/.gimme/versions/go1.10.3.src/src/runtime/panic.go:502 +0x229
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: panic(0x14ae1e0, 0x421b2e550)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/.gimme/versions/go1.10.3.src/src/runtime/panic.go:502 +0x229
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: main.(*TaskRun).uploadArtifact(0x4219e8a00, 0x1607ae0, 0x4220867e0, 0x0)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/gopath/src/github.com/taskcluster/generic-worker/artifacts.go:464 +0x1096
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: main.(*TaskRun).uploadLog(0x4219e8a00, 0x155c9c7, 0x1c, 0x421d2afc0, 0x1f, 0x1)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/gopath/src/github.com/taskcluster/generic-worker/artifacts.go:406 +0x12f
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: main.(*TaskRun).Run.func2(0x42209c080, 0x4219e8a00, 0x42209c088)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:1127 +0xe6
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: panic(0x14ae1e0, 0x421de8ab0)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/.gimme/versions/go1.10.3.src/src/runtime/panic.go:502 +0x229
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: main.(*TaskRun).uploadArtifact(0x4219e8a00, 0x1607ae0, 0x421b03200, 0x1c)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/gopath/src/github.com/taskcluster/generic-worker/artifacts.go:464 +0x1096
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: main.(*TaskRun).Run.func4(0x4219e8a00, 0x42209c080)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:1224 +0x23c
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: main.(*TaskRun).Run(0x4219e8a00, 0x4221ab4a0)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:1285 +0x17c6
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: main.RunWorker(0x0)
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:670 +0xe14
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: main.main()
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: #011/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:418 +0x846
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: 2018/11/16 19:37:03  *********** PANIC occurred! *********** 
Nov 16 19:37:03 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: 2018/11/16 19:37:03 WORKER EXCEPTION due to response code 401 from Queue when uploading artifact &main.S3Artifact{BaseArtifact:(*main.BaseArtifact)(0x421bf2100), Path:"logs/localconfig.json", ContentEncoding:""} with CreateArtifact payload {"contentType":"application/json","expires":"2018-11-30T13:30:39.379Z","storageType":"s3"}
Nov 16 19:37:04 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: 2018/11/16 19:37:04 Exiting worker with exit code 69
Nov 16 19:37:04 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: 2018/11/16 19:37:04 Loading generic-worker config file '/etc/generic-worker.config'...
Nov 16 19:37:04 t-yosemite-r7-265.test.releng.mdc1.mozilla.com generic-worker: 2018/11/16 19:37:04 Creating file /etc/generic-worker.config...

Comment 2

[Automatic Bug Generator]

The worker is now in working state