644402 - Try harder to get a fast slave

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[Nick Thomas [:nthomas] (UTC+12)]

When there are lots of checkins we are often getting VMs for win32 compiles, which are much slower to complete the job. We need to try harder to find a fast slave before falling back to a VM.

I think what is happening is
* one of the masters polls the db for pending work (every minute)
* finds some
* checks slave availability
* doesn't have any fast builders, and so gives work to a VM

That makes sense if we lose a colo or something similarly disastrous, but not for normal operation. I think we need to add some sort of delay, so that work is only assigned to VMs after all the masters have had a chance to poll and assign it to an available fast slave.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This is a dupe of another bug that catlee's working on, but I can't find it right now..

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Perhaps you're thinking of bug 636101, but that's a bit different.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Fom discussion with catlee here's a sketch of how this might work. In _nextFastSlave (http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#286) we can get the list of build requests for the builder (but not the actual request(s) looking for a slave). If we have a fast slave there's no change. If we only have slow slaves, then we only return one if the oldest request is older than some threshold (say 5 minutes). Otherwise we claim there are no slaves available.

Comment 4

[Aki Sasaki (not active)]

Is this still an issue?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

We don't use Windows VMs any more, but we have AWS instances for linux instead. Bug 936222 is the most recent bug to handle this issue.