584289 - Rebalance 10.6 builder minis between production and try pools to match load

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

I'm filing this in RelEng while we sort out how many machines to upgrade, and if we should rebalance the moz2 and try pools. This will get moved to Server Ops when that is decided.

To get us started:
* we have 40 Snow Leopard machines in the moz2 pool. Since 2010-07-13 we have at most 20 concurrent builds running across all branches in that pool. 
* we have 26 Snow Leopard machines in the try pool. Since 2010-07-13 we have at most 25 concurrent builds running, four times above 20.

This suggests to me that we should upgrade all the machines we have, and split them 30/36 between moz2/try (10 above the peak on each pool). 

I'd like to check what proportion of those jobs are tests, since those have mostly moved to the minis. And we also have another 10 machines which already have 2G of RAM which can be re-purposed from 10.5 based builds (assuming load is OK there). Would appreciate your input Messrs Catlee and Hearsum.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

bug#584820 filed to purchase 3gb ram for each of 81 minis being used as osx10.6 builder.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

This bug was about 2 topics a) upgrading ram on 10.6 builder minis, and b) rebalancing builder pools to match workload. 

As IT are doing the ram upgrades in bug#584820, I've morphed this bug to just be about the remaining "rebalance pools" work, which is a RelEng task.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

On the most recent heavy day on try, Fri 8 Oct, we did
snowleopard: 123
  0:      121    98.37%
 15:        2     1.63%

The previous day on non-try, we did 
snowleopard: 105
  0:      105   100.00%

so there isn't any issue at the moment.

Comment 4

[Armen [:armenzg]]

Shall we close this and file a new one if needed?

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #3)
> On the most recent heavy day on try, Fri 8 Oct, we did
> snowleopard: 123
>   0:      121    98.37%
>  15:        2     1.63%
> 
> The previous day on non-try, we did 
> snowleopard: 105
>   0:      105   100.00%
> 
> so there isn't any issue at the moment.

Cool.

(In reply to comment #4)
> Shall we close this and file a new one if needed?
Yep.