1142788 - docker-worker: Adjust polling delay by time to end of billing cycle

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Jonas Finnemann Jensen (:jonasfj)]

### Background: Worker Shutdown Strategy
The worker shutdown when following conditions are met:
 i)   The worker is not executing any tasks, and
 ii)  The worker is close to the end of its billing cycle

Note that the worker may have a capacity=8 meaning that it can run 8
tasks in parallel, but condition (i) is unsatisfied if there is at
least one task is running.

Reasoning:
  (i)  No need to kill if running anything, then we want to finished it,
       besides if we're running one task all system resources will be put
       into this task. So _ideally_ it'll run faster.
  (ii) We'll paying to end of billing cycle, so why shutdown earlier.
  

### Problem: Worker Starvation in Unlikely
Because of (ii) the starvation have to happen at a very specific point
in time. And as messages from azure queue are distributed at random,
when all nodes poll all the time, there is a reasonably risk we'll have one
when (ii) is satisfied.

Example:
  - 500 nodes with capacity 8 are a running.
  - At any given time we're running 4000 tasks.
In this case we're unlikely to be killing nodes, they won't all have 4 tasks
but on average that'll be the case. And likely there will be few nodes with
8 tasks and few nodes with 0 tasks.

Hence, we won't scale down, resulting in a system with a utilization
around 50%.

Granted any tasks running on a system capacity 8, but only 1 task has a
good chance of running faster, so it's 50% of the capacity that isn't
utilized, which is not necessarily the same as 50% of the CPU power.


### Proposed Solution: Adjust Delay between Polling
The polling logic works as follows: (roughly)

  while availableCapacity > 0:
    pollAzureQueuesAndClaimTasks()
    sleep(1s)

Now if instead we do:

  while availableCapacity > 0:
    pollAzureQueuesAndClaimTasks()
    if availableCapacity == totalCapacity && minutesToBillCycleEnd < 10 min:
      sleep(Math.min(10s * (10 - timeToBillCycleEnd), 60s))
    else:
      sleep(1s)

Then if we have more capacity then tasks, idle nodes close the end of their
billing cycle is less likely to pick up new tasks. And if there is plenty of
tasks, they'll get tasks too... I'll just take a little more time.

Note: 10min is an arbitrary number, it should probably be:
  Math.max(10 min, average_task_completion_time)

Where average_task_completion_time is only for tasks that ran on this specific
instance of docker-worker since it started. We can keep a running average
by maintaining: total_task_completion_time and number_of_tasks_completed

### Question: Is this crazy?

Well, is this crazy?
I my probability logic flawed?

Does anyone have better ideas?
The goal is to scale down, preferably while satisfying both (i) and (ii).

Note, the queue can tell us how many pending tasks there is for a given
worker type. I'm not sure how we would use that, but it could be used.
Or maybe the provisioner could have an end-point worker could ping when
they near the end of billing cycle to see if they should stop accepting task.
This way provisioner logic could be employed to decide graceful scale-down.

---
@pmoore, I don't think this is relevant for generic worker. I just wanted to
encourage you to join the discussion :)

Comment 1

[Greg Arndt [:garndt]]

This was implemented in https://github.com/taskcluster/docker-worker/commit/1d23411a82c979f7113122072b37a9f7bc05cf2d