733479 - kickstart DL120G7's in scl1

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

these need to be kickstarted with
 centos6-releng-pupppetagain PUPPET_PASS=...

Before we can do that, we'll need to set their hostnames in DNS.  I suggest
 linux64-hp-slaveNNN
in keeping with the $os-$hw-slave$zeropad pattern.  This is also blocked on getting kickstart and DSPC to play nice in scl1, but that should be easy.

After that, they should come right up, and we can do the rest with puppet.

I *may* want to do bug 732596 first (to install Centos-6.2 instead of 6.0).  I need to figure out how difficult it is to do a "live upgrade" of an already-installed system.  Pointers welcome!

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Yep, let's wait for 732596.

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Dustin J. Mitchell [:dustin] from comment #0)
> Before we can do that, we'll need to set their hostnames in DNS.  I suggest
>  linux64-hp-slaveNNN

what about bld-centos6-hp-NNN ?  linux64 isn't very specific and 'slave' doesn't really add any useful information.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

That naming scheme works for me.  Can you promulgate that within releng as a good standard form to work toward?

These will also need to get added to the puppet node list -- john can help with that if I'm not around.  Regexps are your friend there :)

The kickstart line will actually be
  centos62-releng-pupppetagain PUPPET_PASS=...
(and I still need to copy that KS stuff to admin1a, so it won't work just yet -- but it works in mtv1)

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

This should be good to go in scl1.  Should.

Jake, if you can get DHCP turned over to kickstart before Thursday afternoon, I can get this tested out in scl1 so we know it will work while I'm away.

Comment 5

[Jake Watkins [:dividehex]]

(In reply to Dustin J. Mitchell [:dustin] from comment #4)
> This should be good to go in scl1.  Should.
> 
> Jake, if you can get DHCP turned over to kickstart before Thursday
> afternoon, I can get this tested out in scl1 so we know it will work while
> I'm away.

Dustin: next-server has been flipped to admin1a.  Test away.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Test successful.  You should be good to kickstart these, once they are renamed in DNS and added to http://hg.mozilla.org/build/puppet/ manifests/nodes.pp.

Comment 7

[Amy Rich [:arr] [:arich]]

The hostnames of the dl120s have been changed in dns, dhcp, and inventory.  They are:

bld-centos6-hp-001.build.scl1.mozilla.com.
...
bld-centos6-hp-042.build.scl1.mozilla.com.

They also have aliases in build.mozilla.org.

Comment 8

[Amy Rich [:arr] [:arich]]

With jhford's help, I tracked down a change that needed to be made to the CentOS-6.2-x86_64-releng-puppetagain.cfg file (basically the same fix as https://hg.mozilla.org/build/puppet/rev/717c652a872b).

I've kickstarted bld-centos6-hp-008 and confirmed that I can log in as cltbld.  jhford, can you take a look around and confirm that things look right?

Comment 9

[Amy Rich [:arr] [:arich]]

I've kickstarted the rest of the machines that I could.

bld-centos6-hp-001, bld-centos6-hp-002, bld-centos6-hp-003, bld-centos6-hp-004, bld-centos6-hp-005, bld-centos6-hp-006 can not reach the DHCP server.  Once they cna, they all need to be kickstarted

bld-centos6-hp-028 has an incorrect password for the ilo (it will also need the ilo upgraded when it's reachable), and then the machine needs to be kickstarted.

bld-centos6-hp-031's ilo is unresponsive and needs to be reset, and then the machine needs to be kickstarted.

All other hps (001 - 042, except those mentioned above) in scl1 have been kickstarted with the centos62-releng-puppetagain profile.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

They were, in fact, in the wrong VLAN, due to a switch bug.  Yay!

bld-centos6-hp-{001..006} are now kickstarted.  They'll just need to be rebooted to start the buildslaves (the other machines, kickstarted before bug 733539 landed, will need a puppet run and *then* a reboot)

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Back to the pool for 028 and 031.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Oh, those already have handy-dandy colo-trip bugs.

Comment 13

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Dustin J. Mitchell [:dustin] from comment #12)
> Oh, those already have handy-dandy colo-trip bugs.

bug#735815 and bug#735817