docker-worker: Shutdown inside `idle` event handler



4 years ago
a year ago


(Reporter: jonasfj, Unassigned)



(Whiteboard: [docker-worker])

See comment here:

Note, this could be a non-bug, so correct me if I'm wrong.

From the code it looks like we set a shutdown time, then clear it when work arrives. This means that shutdown can occur while we're in the process of calling queue.claimTask. That would be unfortunately.

I suggest we use an interval `earliestShutdown` and `latestShutdown`.
s.t. if we get an `idle` event while the `reminder` of the billing cycle is:
  earliestShutdown <= reminder <= latestShutdown
We shutdown.

This way, we shutdown no ealier than `earliestShutdown` number of seconds
from end of billing cycle. And if we're closer to the edge of the billing
cycle than latestShutdown, we don't shutdown, because we might already have paid for the next cycle.

Decent defaults for AWS would be:
  earliestShutdown:   6 min
  latestShutdown:     2 min

This means that we must force end polling for tasks in less than ~3 min.
That seems reasonable, ideally we should also enforce that. If we have DNS
issues requests could potentially take longer and node would fail to shutdown.
Component: TaskCluster → Docker-Worker
Product: Testing → Taskcluster
Whiteboard: [docker-worker]
Component: Docker-Worker → Worker
I believe this is wontfix since we will move to tc-worker in the coming quarters.
Last Resolved: a year ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.