1601746 - linux64 hw workers stopped work

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Posix OS, defect)

Description

[:dhouse]

I thought moonshot cartridges were hanging, but a task failure caused workers to stop generic-worker without rebooting.

Comment 1

[:dhouse]

The last task log showed:

[taskcluster 2019-11-28T16:06:01.713Z] Worker Type (releng-hardware/gecko-t-linux-talos) settings:
[taskcluster 2019-11-28T16:06:01.713Z]   {
[taskcluster 2019-11-28T16:06:01.713Z]     "config": {
[taskcluster 2019-11-28T16:06:01.713Z]       "deploymentId": "",
[taskcluster 2019-11-28T16:06:01.713Z]       "runTasksAsCurrentUser": true
[taskcluster 2019-11-28T16:06:01.713Z]     },
[taskcluster 2019-11-28T16:06:01.713Z]     "generic-worker": {
[taskcluster 2019-11-28T16:06:01.713Z]       "go-arch": "amd64",
[taskcluster 2019-11-28T16:06:01.713Z]       "go-os": "linux",
[taskcluster 2019-11-28T16:06:01.713Z]       "go-version": "go1.10.8",
[taskcluster 2019-11-28T16:06:01.713Z]       "release": "https://github.com/taskcluster/generic-worker/releases/tag/v14.1.2",
[taskcluster 2019-11-28T16:06:01.713Z]       "revision": "13118c4c1ba10f863f39d6c623b3dd59ca6e0f00",
[taskcluster 2019-11-28T16:06:01.713Z]       "source": "https://github.com/taskcluster/generic-worker/commits/13118c4c1ba10f863f39d6c623b3dd59ca6e0f00",
[taskcluster 2019-11-28T16:06:01.713Z]       "version": "14.1.2"
[taskcluster 2019-11-28T16:06:01.713Z]     },
[taskcluster 2019-11-28T16:06:01.713Z]     "machine-setup": {
[taskcluster 2019-11-28T16:06:01.713Z]       "config": "https://github.com/mozilla-releng/build-puppet/raw/master/modules/generic_worker/templates/generic-worker.config.erb",
[taskcluster 2019-11-28T16:06:01.713Z]       "docs": "https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Modules/generic_worker"
[taskcluster 2019-11-28T16:06:01.713Z]     }
[taskcluster 2019-11-28T16:06:01.713Z]   }
[taskcluster 2019-11-28T16:06:01.713Z] Task ID: NZFadBr7QqWG0E6CiuNcmA
[taskcluster 2019-11-28T16:06:01.713Z] === Task Starting ===
[taskcluster 2019-11-28T16:06:01.788Z] [mounts] Moving existing writable directory cache gecko-level-3-checkouts from /home/cltbld/caches/QRfghPYrSpaBUGxOTa4GMw to /home/cltbld/tasks/task_1574954327/home/cltbld/checkouts
[taskcluster 2019-11-28T16:06:01.788Z] [mounts] Creating directory /home/cltbld/tasks/task_1574954327/home/cltbld with permissions 0700
[taskcluster:error] goroutine 1 [running]:
[taskcluster:error] runtime/debug.Stack(0xc420d47430, 0x8eb7c0, 0xc420b7ea70)
[taskcluster:error]     /home/travis/.gimme/versions/go1.10.8.linux.amd64/src/runtime/debug/stack.go:24 +0xa7
[taskcluster:error] main.(*TaskRun).Run.func2(0xc4200a2088, 0xc420106300, 0xc4200a2090)
[taskcluster:error]     /home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:949 +0x22a
[taskcluster:error] panic(0x8eb7c0, 0xc420b7ea70)
[taskcluster:error]     /home/travis/.gimme/versions/go1.10.8.linux.amd64/src/runtime/panic.go:502 +0x229
[taskcluster:error] main.(*WritableDirectoryCache).Mount(0xc4218c7bd0, 0xc420106300, 0x0, 0x0)
[taskcluster:error]     /home/travis/gopath/src/github.com/taskcluster/generic-worker/mounts.go:444 +0xa02
[taskcluster:error] main.(*TaskMount).Start(0xc421cb5ea0, 0x1b)
[taskcluster:error]     /home/travis/gopath/src/github.com/taskcluster/generic-worker/mounts.go:361 +0x1bd
[taskcluster:error] main.(*TaskRun).Run(0xc420106300, 0xc4218c6190)
[taskcluster:error]     /home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:1025 +0x1386
[taskcluster:error] main.RunWorker(0x0)
[taskcluster:error]     /home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:494 +0xc84
[taskcluster:error] main.main()
[taskcluster:error]     /home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:186 +0x926
[taskcluster:error]
[taskcluster:error] &errors.errorString{s:"[mounts] Not able to rename dir /home/cltbld/caches/QRfghPYrSpaBUGxOTa4GMw as /home/cltbld/tasks/task_1574954327/home/cltbld/checkouts: rename /home/cltbld/caches/QRfghPYrSpaBUGxOTa4GMw /home/cltbld/tasks/task_1574954327/home/cltbld/checkouts: no such file or directory"}
[taskcluster:error] [mounts] Not able to rename dir /home/cltbld/caches/QRfghPYrSpaBUGxOTa4GMw as /home/cltbld/tasks/task_1574954327/home/cltbld/checkouts: rename /home/cltbld/caches/QRfghPYrSpaBUGxOTa4GMw /home/cltbld/tasks/task_1574954327/home/cltbld/checkouts: no such file or directory

Comment 2

[:dhouse]

I rebooted the machines in this state and they recovered.