489333 - Partition slave pools to improve throughput by caching and less clobbering

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Axel Hecht [:Pike]]

We talked about this in yesterday's l10n build call, I thought I should get a bug on file:

It probably makes sense to partition the slave pools by task, so that we have a higher likelyhood of running the same builds on the same slaves, and overall get lower disk usage, and thus better depend builds.

The partitions shouldn't be mutually exclusive, nor should the slave setup become specialized. Just partition the setup such that we have two or three kinds of builds per slave instead of N.

For example, we currently have 26 slaves for 1.9.1 windows l10n builds. Looking at the rough data, a build takes 4 minutes, 1 minute of that is locale dependent. Assuming that we can cache most of the 3 minutes, let's say 2 minutes setup, 2 minute per locale. That's getting us an optimal throughput with 8 slaves for 70 locales concurrently building, or to be precise square root of locales count. On linux, setup time is more in the order of 80%, which leads to half the amount of slaves as on windows for optimal throughput, aka 4. Check my math, I came up with number of slaves is 
sqrt( (1-s)/s * n), 
where s is the setup time fraction (0.8 for linux, 0.5 for win), and n is the number of concurrent build requests.

Another metric to allocate slaves to pool fragments by is how much disk size the builds need, and how much disk space we have (either right now, or when we beef up the storage even further). Optimally, we wouldn't have to clobber builds for disk constraints.

Activity on trees is another one, like, the 70 builds for locales are actually "how many concurrent build requests should the pool handle". That probably leads to allocation of smaller fragments to feature branches or stable releases.

Comment 1

[Chris AtLee [:catlee]]

Great idea to do this for l10n builds.  There's a patch over in bug #488411 that I think does this.

I'm not sure if this would help or hurt us for our other types of builds.

Comment 2

[Chris AtLee [:catlee]]

Done for l10n builds as part of bug #488411.