Closed Bug 1421490 Opened 7 years ago Closed 6 years ago

docker-worker seems to not resolve exception w. worker-shutdown if gracefully terminated

Categories

(Taskcluster :: Workers, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: jonasfj, Assigned: wcosta)

References

Details

If the machine is gracefully terminated, like EC2 inititated shutdown (when updating AMI),
it seems workers don't resolve with exception and worker-shutdown.


Instead it seems the SIGTERM reaches into docker and kills all task container processes and then containers, causing docker-worker to conclude the task failed because it got a non-zero exit code.
For shutdown behavior, see: https://forums.aws.amazon.com/message.jspa?messageID=92493#92493

In short, it'll hit the power button, which usually triggers graceful machine shutdown.
After a short grace period the machine power cord is unplugged, if it haven't already halted.
So I know we used to do this right.
Because in the days before we got spot-termination notices, we relied exclusively on this.

The problem is either:
  A) when the worker process is terminated, it ignores the signal rather than reporting tasks 'worker-shutdown'
     (even just silently dying would be fine; I suspect it's actively ignoring the signal)
  B) docker receives the terminate signal before the worker process does (maybe there is some race condition)
Assignee: nobody → wcosta
See Also: → 1438128
See Also: 1438128
Priority: -- → P2
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
Component: Docker-Worker → Workers
You need to log in before you can comment on or make changes to this bug.