577154 - Add 15 linux64 build slaves to production pool

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

After the VMs are created in bug#577152, this bug is to track any manual setup we need to do in the slaves, and the matching buildbot masters setup.

Comment 1

[Armen [:armenzg]]

Dependent bug as been WONTFIXED due to lack of ESX space.

Let's think of repurposing VMs from production.

Comment 2

[Chris AtLee [:catlee]]

Assigning to you John, to figure out where these machines should come from.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #2)
> Assigning to you John, to figure out where these machines should come from.

We've got 42 ix machines now powered and racked in internap. Once we get the refimage setup (in bug#588957), and these machines imaged, then we can use this bug to track putting these ix slaves into production.

Nothing more for me to do here, so putting back in the pool.

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

We have 39 linux64 slaves.  How many should be in production, staging and try?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

My 2 cents: 3-5 for staging, split the rest evenly between prod and try. 

We could push more to try given they're always clobber builds, and try is about 50% of the pushes. The number of prod pushes will probably go up as more project branches get going though, and no way of knowing how much. 50/50 isn't a bad guess.

Comment 6

[Armen [:armenzg]]

I was going to say the following but I guess we can shuffle later if needed.

> How does 3 on staging, 65% on try and 35% on mozilla-central sound like?
> 
> I say 65% instead of 50% (~% of monthly load) because try builds are clobber type.
> You could use the reports that anamarias wrote but the last 2-3 months do not
> have a normal/standard development load as we devs were blocked by FF4 and were
> using try much more.

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

summary of discussion with jhford. We're reduced from 42 ix machines because of "borrowed" machines for extra masters, and hardware failures, so only have 39 available to use:

To start with, lets allocate these as follows:
01 - refimage
02 - staging
18 - production builders
18 - tryserver builders


After these are deployed and we monitor load/wait times, we can rebalance pools as needed.

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

> 01 - refimage
linux64-ix-ref
> 02 - staging
linux64-ix-slave01, 02
> 18 - production builders
linux64-ix-slave03-21
> 18 - tryserver builders
linux64-ix-slave22-40

Comment 9

[John Ford [:jhford] CET/CEST Berlin Time]

probably should have uploaded https://bugzilla.mozilla.org/attachment.cgi?id=522767&action=diff to this bug, but this is the patch to add these slaves to the buildbot configs

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

Status update:

bm1,bm2 correspond to buildbot-master%i.build.scl1.mozilla.com

linux64-ix-slave03.build.mozilla.org    bm1:9010
linux64-ix-slave05.build.mozilla.org    bm1:9010
linux64-ix-slave06.build.mozilla.org    bm1:9010
linux64-ix-slave07.build.mozilla.org    bm1:9010
linux64-ix-slave08.build.mozilla.org    bm1:9010
linux64-ix-slave09.build.mozilla.org    bm1:9010
linux64-ix-slave15.build.mozilla.org    bm2:9010
linux64-ix-slave17.build.mozilla.org    bm2:9010
linux64-ix-slave18.build.mozilla.org    bm2:9010
linux64-ix-slave19.build.mozilla.org    bm2:9010
linux64-ix-slave20.build.mozilla.org    bm2:9010
linux64-ix-slave21.build.mozilla.org    bm2:9010

Are all in production.

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

The remaining try machines are being tracked in bug 647049