1154276 - [docker-worker] Prevent tasks from being claimed when instance is shutting down

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Greg Arndt [:garndt]]

There have been many occasions where docker-worker will be killed by the system because a spot node is being terminated, but then the worker will respawn right before shutdown (after syslog has been killed), claim a task, but then the whole system goes down.  This causes the task to be claimed for up to 20 minutes (takenUntil) without being retried by another worker.

On shutdown, perhaps there is a way to cause the worker to not respawn, or if it does start up, prevent it from claiming tasks.

Comment 1

[Pete Moore [:pmoore][:pete]]

I think the runlevel command will tell you if your system is currently being shutdown, maybe this could be used to check whether to respawn the worker?

Comment 2

[Greg Arndt [:garndt]]

Discussed on IRC and it might be possible to instruct the docker-worker to wait for docker and only start up when run levels are 2-5.

Also, it will stop when docker stops or when run levels are not 2-5.

Comment 3

[Greg Arndt [:garndt]]

Attachment: GH PR 80

This is the PR that we were discussing previously.  Added some notes to it to explain what it's doing.

Comment 4

[Jonas Finnemann Jensen (:jonasfj)]

Comment on attachment 8595997 [details] [review]
GH PR 80

Gave a light review... Looks good to me.
Didn't read the test cases, but the code changes looks good.
Also didn't focus too much on the refactored parts. But I like the cleanup :)

Comment 5

[Greg Arndt [:garndt]]

https://github.com/taskcluster/docker-worker/commit/025a39472e9f21c02075421ee6ef76e632010c97