1517571 - Increase maxCapacity for android-components-g worker type

Status: RESOLVED FIXED

(Taskcluster :: Operations and Service Requests, task)

Description

[:Grisha Kruglov]

Currently the maxCapacity is set at 12 iirc; our most common use case of the worker type schedules around 60-70 tasks at a time (for a GitHub PR push), so it would be great to have a higher capacity to churn through them quicker.

I'm not sure what the guidelines are for how much we can request. At least double the current capacity would be nice.

Comment 1

[:Grisha Kruglov]

Individual tasks do take a few minutes (up to about 13 minutes currently, mostly around 7-9minutes) to run, so with, say, two PRs pushed at the same time, we'll have about 150 tasks. Ideally I'd like to have maxCapacity high enough (scale up to 150 workers? allow workers to execute more than a single job at a time?) to be able to handle that case with aplomb.

Comment 2

[Johan Lorenzo [:jlorenzo]]

More details in https://github.com/mozilla-mobile/android-components/pull/1638#pullrequestreview-189314008

Comment 3

[Johan Lorenzo [:jlorenzo]]

I just set the max capacity to a 100.

Like said in the Github comment above, we have that same max capacity on gecko-3-b-android[2]. This worker type covers mozilla-central, mozilla-beta, and mozilla-release. The workload is not the same, we usually have a couple of pushes on each repo a day.
gecko-1-b-android[3] is set at 1000, to cope with try. We probably want a number between 100 and 1000. I don't have much experience on managing these numbers. John, Pete, what are your takes on this? 


[1] https://tools.taskcluster.net/aws-provisioner/android-components-g/view
[2] https://tools.taskcluster.net/aws-provisioner/gecko-3-b-android/view
[3] https://tools.taskcluster.net/aws-provisioner/gecko-1-b-android/view

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Johan Lorenzo [:jlorenzo] from comment #3)
> John, Pete, what are your takes on this? 

I don't have info on how big the pool should be to service capacity, but from a provisioning perspective, if they're using a common instance type from Amazon, then we should be fine with any number between 100-1000.  The key is to watch for provisioning errors and make sure that if there are a lot of failures encountered that the number is lowered a little.

Comment 5

[Johan Lorenzo [:jlorenzo]]

Okay, great! Thanks for the context!

I just bumped it to 500. When I did some of my tests alone, I was close to the 100 limit.