backlogged gecko-t/win11-64-2009 pool
Categories
(Release Engineering :: Firefox-CI Administration, defect)
Tracking
(Not tracked)
People
(Reporter: jcristau, Assigned: jmoss)
Details
Attachments
(2 external files)
Looking at https://sql.telemetry.mozilla.org/queries/96140?p_provisionerid=gecko-t&p_workerType=win11-64-2009#237445, it seems the time tasks wait to get picked up regularly exceeds an hour, sometimes much more than that. Can we look at increasing maxCapacity (currently 600), or would that cause other issues?
|
||
Comment 1•2 years ago
|
||
jmoss: looking at the number of regions this worker pool uses, we can probably safely double the maxCapacity.
Pushed by jmoss@mozilla.com:
https://hg.mozilla.org/ci/ci-configuration/rev/66e48811551b
Decrease worker idle timeout to 300 seconds + bump max capacity for gecko-t/win11-64-2009 pool. r=relsre-reviewers,markco
|
||
Comment 4•2 years ago
|
||
Would there be a drawback in increasing the max capacity even further? I think that merely doubling isn't enough when at peak demands.
Thanks
|
Reporter | |
Comment 5•2 years ago
|
||
I'm not sure about azure, but on gcp one drawback would be increasing the chance of running into quota limits and starving other pools.
I can request an increase in our spot quota for all of the regions. Right now, all of our windows 11 cloud workers share a spot quota of 3200 vCPU per region, so increasing max capacity in gecko-t/win11-64-2009 may cause issues in other worker pools in the same region, while also potentially hitting the quota limit. You can find this information in worker-pools.yml in the spot: true config.
There is also some efforts in improving the azure worker startup time between taskcluster's worker manager and the configuration we run on initial boot on the windows worker. Ultimately, increasing the max capacity is not the only way to increase the backlog of workers. It's something Release SRE is working on.
|
|
||
Comment 8•2 years ago
|
||
The question we should ask is, what is our typical max demand? We should take that number and add 10%. Then work backward to figure out the setup to accommodate it. There will be a limit. Like, it may not be practical to run 10000 VMs at one time, but let's look at the need and build from there.
We can add more regions and as well as asked for quota increases. Also, worker-pools can be isolated to regions to prevent any contention between pools.
|
||
Comment 10•2 years ago
|
||
|
||
Comment 11•2 years ago
|
||
A patch has been attached on this bug, which was already closed. Filing a separate bug will ensure better tracking. If this was not by mistake and further action is needed, please alert the appropriate party. (Or: if the patch doesn't change behavior -- e.g. landing a test case, or fixing a typo -- then feel free to disregard this message)
Description
•