reduce worker idle time to 0 for test workers
Categories
(Taskcluster :: General, enhancement)
Tracking
(Not tracked)
People
(Reporter: jmaher, Unassigned)
Details
analyzing data over the last week about bootstrap time and idle times, it seems we can see 8-18% total cpu reduction by terminating a test worker when it finishes a task and there are no tasks currently in the queue. I propose:
when finished the 'last' task, query the queue for more work, if none, shutdown fast
these worker pools:
generic-worker:
- t-win10-64
- t-win10-64-gpu-s
- t-win7-32
- t-win7-32-gpu
docker-worker:
- t-linux-large
- t-linux-xlarge
we will monitor this to verify calculations are accurate.
my understanding is we can configure for linux:
https://hg.mozilla.org/ci/ci-configuration/file/tip/worker-pools.yml#l21
not sure if we can do that for specific workers, but that would make the most sense. I think builders and source checkouts will have a longer bootstrap cycle.
I am not sure how to get windows to not do the final reboot prior to the idle timeout.
|
Reporter | |
Comment 1•6 years ago
|
||
:bstack- do you think we can override the timeout value for t-linux-large and t-linux-xlarge ?
:pmoore- can you outline what needs to be done on the windows machines that run generic-worker in order to terminate if the queue is empty instead of reboot and wait for idletimeout?
|
||
Comment 2•6 years ago
|
||
I think the values in that file are just defaults so it should be possible to change specific workerpools. Releng ows ci-configuration so they would know the most about how they want you to go about changing those values.
|
|
Reporter | |
Comment 3•6 years ago
|
||
:tomprince, any concerns with changing https://hg.mozilla.org/ci/ci-configuration/file/tip/worker-pools.yml#l21 to be specific for t-linux-large and t-linux-xlarge?
|
||
Comment 4•6 years ago
|
||
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #3)
:tomprince, any concerns with changing https://hg.mozilla.org/ci/ci-configuration/file/tip/worker-pools.yml#l21 to be specific for t-linux-large and t-linux-xlarge?
I assume you mean overriding them for those worker types? I don't have any concerns.
|
|
Reporter | |
Comment 5•6 years ago
|
||
thanks, that is what I was referencing
|
|
Reporter | |
Comment 6•6 years ago
|
||
:catlee, could you weigh in on how the workers would need to call the taskcluster api to query/claim a task? My original understanding was not correct and I want to make sure that it is well documented here.
I assume we will need to change both docker-worker and generic-worker.
|
||
Comment 7•6 years ago
|
||
The workers claim only one task at a time. After they finish a task, they could call the "pendingTasks" endpoint to check if there are any tasks potentially available before rebooting or shutting down: https://docs.taskcluster.net/docs/reference/platform/queue/api#pendingTasks
There's no guarantee that there will be tasks to claim after the worker finishes rebooting, but it's better than rebooting and then shutting down immediately.
I think both docker-worker and generic-worker would need to be changed with this logic.
|
||
Comment 8•6 years ago
|
||
(In reply to Chris AtLee [:catlee] from comment #7)
The workers claim only one task at a time. After they finish a task, they could call the "pendingTasks" endpoint to check if there are any tasks potentially available before rebooting or shutting down: https://docs.taskcluster.net/docs/reference/platform/queue/api#pendingTasks
There's no guarantee that there will be tasks to claim after the worker finishes rebooting, but it's better than rebooting and then shutting down immediately.
I think both docker-worker and generic-worker would need to be changed with this logic.
If it is feasible, maybe we should take into account unscheduled tasks (whose dependencies have not failed) too, since they might be scheduled soon.
|
|
Reporter | |
Comment 9•6 years ago
|
||
that would probably mean a change to the api, who knows if that is simple. I think for this bug we fix the test machines to call pendingTasks, then reboot asap. We can follow up with a better algorithm at the api endpoint to predict load.
|
||
Comment 10•6 years ago
|
||
I think there are potentially much bigger wins to be made with optimised provisioning logic now that we have a postgres backend. Historically we burned a lot of resources because we kept provisioning instances, but because they took a long time to start up, far more instances were created than needed in response to demand. Many of these instances started up, and had nothing to do, and eventually shut down after a period of inactivity. This was mostly a limitation of our Azure back end, because we couldn't see inside the pending task queues to know what was acutally pending, and whether we had dealt with it already, and we didn't track when workers became available for work.
Both these issues have been solved now, so probably the biggest gains to be made are by capitalising on these new abilities.
Regarding shutting down machines rather than rebooting and then shutting down, I'm not sure how much that saves vs the engineering effort to implement, but if it is decided to be beneficial, probably the best place for that to be done is in worker-runner, rather than in the workers directly.
|
|
Reporter | |
Comment 11•6 years ago
|
||
pmoore, shutting down asap (including the cost of additional instances to boot in worse case scenario) will save 10% of total hours on win* workers. total hours != billed hours because of orphaned workers.
I am not familiar with worker-runner, what is that? what type of effort would be required to fix the generic-worker for windows to shutdown asap?
|
|
Reporter | |
Updated•6 years ago
|
|
|
||
Updated•6 years ago
|
|
||
Comment 12•6 years ago
|
||
Dialing the idle-waiting-for-task time down is probably a good idea, and easy enough to do in the worker pool definitions. So I'll comment only on the halt-without-rebooting suggestion in comment 7.
I'm surprised to hear that this additional reboot before halting is a big cost.
I see docker-worker mentioned above. Docker-worker doesn't reboot, so it's not relevant here.
The check-pending-tasks approach is a feasible idea, although it has some downsides such as those marco has pointed out. My understanding of Firefox-CI windows systems is that the process to start up a new instance is substantially more expensive than rebooting, so this probably isn't a win unless the savings in number of reboots is several times higher than the number of additional new instance startups required due to diminished capacity where workers shut down unnecessarily. There's also a consideration of hot caches, which must be re-primed for new instances. So a SWAG is that the number of reboots avoided should be 10x the number of additional instance start-ups, if this change is going to break even.
On firefox-ci workers, my understanding is that workers are wrapped with some batch scripts, rather than with worker-runner, and that's what does the rebooting. So that would be the place to check for pending tasks. That's going to be pretty tricky to implement and test -- ballpark a week's work, and by a team with a lot of other stuff going on.
Another option is to just configure these workers to run one task and shut down. AIUI the vast majority of windows workers run zero or one tasks, so that might be a quick win at the cost of some extra instantiations to replace those workers which would otherwise have run more than one task. This approach is just a configuration option in the worker pool, so could be trialed with little development effort.
|
|
Reporter | |
Comment 13•6 years ago
|
||
bootstrap times for linux are ~1 minute, and windows ~9 minutes.
shutdown times for linux are ~15 minutes, and windows ~6 minutes.
idle timeout for linux is 15 minutes, windows is 90 seconds.
of all hours billed in Jan-Apr 2020 8.38% of them were spent timing out/shutting down after the last task. This is skewed on linux, but still significant on windows.
The windows reboot takes almost 5 minutes between reboot and workerReady- we speculate that if no jobs are available for work after the task is finished that shutting down more workers will allow other workers to pick up the extra job within the next 15 minutes.
even taking worse case scenario of all tasks added to queue after last task finished and incurring the bootstrap, we would still save >5% on all worker types. I am not aware of how to calculate any pre-fetching/caching of instances.
I like the idea of doing 1 or 2 tasks per windows worker- for windows that is 85-90% of all use cases except for win10-gpu which is only 75%.
|
|
||
Comment 14•6 years ago
|
||
shutdown times for linux are ~15 minutes, and windows ~6 minutes.
Can you clarify what "shutdown time" refers to here? I assume it is the amount of time between two events, but I'm not sure if the first event is:
- the resolution timestamp from a resolved task run
- a generic-worker log message from the worker to say that generic-worker is exiting and requires a reboot
- an OCC log message saying it is going to reboot the worker
- a log message from the worker to say it has reached its idle timeout and will shut down
- a different OCC log message
- etc
and the second event:
- the worker is no longer reported by aws ec2 DescribeInstance api call
- some EC2 instance metadata termination time
- the last timestamp logged in a papertrail log from the worker
- etc
Depending on how this time is being measured, there could be several culprits. For example, the worker is configured to remain running for an idle timeout before terminating - this is configurable in generic-worker, and can be reduced to e.g. 1 second if required - so that if no task is available within 1 second of the worker starting up after a reboot, the worker will terminate. If this value was set to e.g. 90s, then the worker would be intentionally idle for 90s after rebooting if there were no tasks.
My first suggestion would be to set generic-worker config setting idleTimeoutSecs to e.g. 1 so that as soon as there are no pending tasks, and the worker has no work to do, it shuts down. Don't set this to 0 as 0 means "wait forever":
idleTimeoutSecs How many seconds to wait without getting a new
task to perform, before the worker process exits.
An integer, >= 0. A value of 0 means "never reach
the idle state" - i.e. continue running
indefinitely. See also shutdownMachineOnIdle.
If you really want to avoid the reboot, the relops scripts that start generic-worker can terminate the machine after one task has run. The reboot/shutdown logic is handled outside the generic-worker code, by OpenCloudConfig, I believe. If OpenCloudConfig is no longer used, I suspect there are some other scripts which start generic-worker and check its exit code. Whatever starts generic-worker up, can terminate the machine when generic-worker exits. The best contacts for discussing these options are the RelOps team.
|
|
Reporter | |
Comment 15•6 years ago
|
||
I measured from timestamps in bigquery which are parsed from papertrail messages.
Basically:
instanceShutown-taskFinished = time_to_shutdown
this includes the idleTimeoutSecs (either 90 or 900 seconds) and on windows a reboot and generic-worker startup. There is a reference to the linux value in comment 3. Good to know that 0 is wait forever!
I will dig into the relops scripts. I need to know:
- how the script determines to reboot?
- how the instance knows to shutdown?
:grenade, do you know a bit about how our windows aws workers reboot, and shutdown?
|
|
||
Comment 16•6 years ago
|
||
I wonder if we're also waiting for an instance to do a "clean" shutdown, and how long that takes, verses a more forceful shutdown.
|
|
Reporter | |
Comment 17•6 years ago
|
||
:catlee, what are you referring to? Is this something that isn't documented in papertrail and overhead at aws? Each worker as overhead that we are not able to account for, typically it is 2-7 minutes, I think the windows workers are ~211 seconds.
|
|
||
Comment 18•6 years ago
|
||
I'm wondering what mechanism we're using to shut down the machine. If for example we're doing a 'poweroff' or its equivalent on Windows, we may be spending time shutting down services on the machine, syncing filesystem caches, etc., which is a waste of time if the instance is going to be deleted after it finishes shutting down.
|
|
||
Comment 19•6 years ago
|
||
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #15)
:grenade, do you know a bit about how our windows aws workers reboot, and shutdown?
in the current configuration on ec2 (we are transitioning to worker runner on azure), generic-worker is started by a wrapper script
when a task completes, generic worker returns an exit code that tells the wrapper script whether it should halt (resulting in the instance termination) or reboot (resulting in the instance polling the queue for more work).
when we have completed the transition to worker runner in azure (imminent), there will be no wrapper script and it will be up to worker runner to manage reboots and halting.
|
|
||
Updated•6 years ago
|
Description
•