1421490 - docker-worker seems to not resolve exception w. worker-shutdown if gracefully terminated

Status: RESOLVED WONTFIX

(Taskcluster :: Workers, defect, P2)

Description

[Jonas Finnemann Jensen (:jonasfj)]

If the machine is gracefully terminated, like EC2 inititated shutdown (when updating AMI),
it seems workers don't resolve with exception and worker-shutdown.


Instead it seems the SIGTERM reaches into docker and kills all task container processes and then containers, causing docker-worker to conclude the task failed because it got a non-zero exit code.

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

For shutdown behavior, see: https://forums.aws.amazon.com/message.jspa?messageID=92493#92493

In short, it'll hit the power button, which usually triggers graceful machine shutdown.
After a short grace period the machine power cord is unplugged, if it haven't already halted.

Comment 2

[Jonas Finnemann Jensen (:jonasfj)]

So I know we used to do this right.
Because in the days before we got spot-termination notices, we relied exclusively on this.

The problem is either:
  A) when the worker process is terminated, it ignores the signal rather than reporting tasks 'worker-shutdown'
     (even just silently dying would be fine; I suspect it's actively ignoring the signal)
  B) docker receives the terminate signal before the worker process does (maybe there is some race condition)