837234 - Repurpose linux ix centos5 slaves as centos6/mock builders

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris AtLee [:catlee]]

We have plenty of spare linux-ix and linux64-ix slaves right now. I think they're only used for ESR10 and ESR17 builds at the moment.

We should re-purpose them as centos6/mock builders.

The re-imaged slaves from bug 837116 are working well in staging so far. One change I had to make to my configs was to remove '-ix' from being considered 'fast' here:
http://hg.mozilla.org/build/buildbot-configs/file/03127b8a1025/mozilla/build_localconfig.py#l80

TBD:
* How many to move?
* What to call them?
* slavealloc changes
* buildbot-config changes
* puppet changes

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

 * Should we put them in build.releng.scl1.mozilla.com instead of build.mozilla.com?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Need to fix this nagios check too:
<nagios-releng> Sat 07:57:59 PST [413] linux64-ix-slave34.build.scl1.mozilla.com:IDE S.M.A.R.T - /dev/sda1 is 4CRITICAL: CRITICAL - SMART_ENABLE: Inappropriate ioctl for device (http://m.allizom.org/IDE+S.M.A.R.T+-+/dev/sda1)

Same for the other slave imaged that was converted, linux64-ix-slave35.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Hm, I suspect we'll actually need to enable smart monitoring - I'll do that in another bug.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, perhaps not:

(CentOS 5)
[root@linux64-ix-slave32 ~]# /usr/lib64/nagios/plugins/check_ide_smart -n -d /dev/sda1
OK - Operational (22/22 tests passed)
[root@linux64-ix-slave32 ~]# /usr/lib64/nagios/plugins/check_ide_smart -n -d /dev/sda
OK - Operational (22/22 tests passed)

(CentOS 6)
[root@linux64-ix-slave35 ~]# /usr/lib64/nagios/plugins/check_ide_smart -n -d /dev/sda1
CRITICAL - SMART_ENABLE: Inappropriate ioctl for device
CRITICAL - SMART_CMD_ENABLE
[root@linux64-ix-slave35 ~]# /usr/lib64/nagios/plugins/check_ide_smart -n -d /dev/sda
OK - Operational (17/17 tests passed)

so it looks like we can just change all of these to use /dev/sda (which makes a whole lot more sense anyway!)

I did that.  Let's see what happens.

Comment 5

[Chris AtLee [:catlee]]

Our centos6 builders are currently named:

bld-centos6-hp-XXX
bld-linux64-ec2-XXX
try-linux64-ec2-XXX

I propose we rename this ix slaves to:
bld-linux64-ix-XXX
try-linux64-ix-XXX (if it's not too much work)

Since Jan 1 we've only done 491 builds on these machines, the bulk of which are on the mozilla-esr17 branch. We could probably keep 2-3 each of the old 32/64 bit slaves to service esr17 for now. That leaves ~70 slaves we could repurpose?

Comment 6

[Chris Cooper [:coop] (he/him)]

Somewhat related: what are we going to do with the mv-moz2-linux-ix slaves that are still in MV? Will they be moved/upgraded/re-imaged as part of this? We only have 32-bit builders in MV AFAICT.

Comment 7

[Chris AtLee [:catlee]]

(In reply to Chris Cooper [:coop] from comment #6)
> Somewhat related: what are we going to do with the mv-moz2-linux-ix slaves
> that are still in MV? Will they be moved/upgraded/re-imaged as part of this?
> We only have 32-bit builders in MV AFAICT.

No reason to wait. We can move/upgrade the mv-based linux ix systems at any time IMO.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Dustin J. Mitchell [:dustin] from comment #1)
>  * Should we put them in build.releng.scl1.mozilla.com instead of
> build.mozilla.com?

no, we won't be changing VLAN.

Comment 9

[Chris AtLee [:catlee]]

Attachment: all linux ix slaves in slavealloc

Comment 10

[Chris AtLee [:catlee]]

Attachment: machines to reimage

of the 85 linux ix slaves in slavealloc, I've left the first 6 for linux32 and linux64. The remaining 73 slaves are listed here.

I think these should be split 50/50 between production and try. So let's take 37 of them and name them bld-linux64-ix-{001..037} and the remaining 36 of them try-linux64-ix-{001..036}

The mv instances should have their RAM upgraded before being put back into the pool, since they only have 4Gb and should have 8Gb.

Comment 11

[Chris AtLee [:catlee]]

Attachment: puppet manifests for new ix slaves

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 721205 [details] [diff] [review]
puppet manifests for new ix slaves

(no need for review for node changes)

Comment 13

[Chris AtLee [:catlee]]

FTR, we're going to name them all bld-linux64-ix-{001..073}. 001..037 will be for production, and 038..073 will be for try.

Comment 14

[Chris AtLee [:catlee]]

Attachment: add new slaves to buildbot

Comment 15

[Chris AtLee [:catlee]]

Attachment: remove distinction between fast/slow slaves

Comment 16

[Chris AtLee [:catlee]]

Added all of the above to slavealloc, and copied ssh keys.

I've put bld-linux64-ix-001 into the production pool.

Comment 17

[Chris AtLee [:catlee]]

Seeing some green there. Putting the rest into the production pool.

Comment 18

[Chris AtLee [:catlee]]

Put the try machines online too

Comment 19

[Armen [:armenzg]]

(In reply to Dustin J. Mitchell [:dustin] from comment #12)
> Comment on attachment 721205 [details] [diff] [review]
> puppet manifests for new ix slaves
> 
> (no need for review for node changes)

17:36 arr: armenzg_dinner: that's only scl1, not mtv1
17:36 arr: so someone needs to add the same for /bld-linux64-ix-\d+.build.mtv1.mozilla.com/
17:37 arr: catlee: ^^

Comment 20

[Rail Aliiev [:rail]]

catlee, can you make sure that xrbld key is a proper one? bld-linux64-ix-008 failed to upload to the symbolpush servers.

Comment 21

[Justin Wood (:Callek)]

(In reply to Armen Zambrano G. [:armenzg] from comment #19)
> (In reply to Dustin J. Mitchell [:dustin] from comment #12)
> > Comment on attachment 721205 [details] [diff] [review]
> > puppet manifests for new ix slaves
> > 
> > (no need for review for node changes)
> 
> 17:36 arr: armenzg_dinner: that's only scl1, not mtv1
> 17:36 arr: so someone needs to add the same for
> /bld-linux64-ix-\d+.build.mtv1.mozilla.com/
> 17:37 arr: catlee: ^^

Done: http://hg.mozilla.org/build/puppet/rev/545d50d04d7f

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

These need post-image setup (eg ssh keys for try):
bld-linux64-ix-051.build.mtv1.mozilla.com
bld-linux64-ix-052.build.mtv1.mozilla.com
bld-linux64-ix-053.build.mtv1.mozilla.com
bld-linux64-ix-054.build.mtv1.mozilla.com

(from https://bugzilla.mozilla.org/show_bug.cgi?id=847529#c9)

Comment 23

[Chris AtLee [:catlee]]

(In reply to Rail Aliiev [:rail] from comment #20)
> catlee, can you make sure that xrbld key is a proper one? bld-linux64-ix-008
> failed to upload to the symbolpush servers.

I dug into this today. It looks like we have at least two generations of xrbld keys in active use. Both are valid for accessing stage, but only the newer one is valid for accessing symbolpush. The bld-linux64-ix slaves were set up with the old key. Additionally, the buildbot masters which have the xrbld key on them (not all do!) are using the old key.

I've copied the new key to bld-linux64-ix-{001..037} (except 004, which is offline ATM) and all the buildbot masters that currently have the xrbld key have been updated.

Comment 24

[Chris AtLee [:catlee]]

(In reply to Nick Thomas [:nthomas] from comment #22)
> These need post-image setup (eg ssh keys for try):
> bld-linux64-ix-051.build.mtv1.mozilla.com
> bld-linux64-ix-052.build.mtv1.mozilla.com
> bld-linux64-ix-053.build.mtv1.mozilla.com
> bld-linux64-ix-054.build.mtv1.mozilla.com
> 
> (from https://bugzilla.mozilla.org/show_bug.cgi?id=847529#c9)

Did these.

I think we're done here.