773331 - Setup 16 lion machines in SCL3

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Chris Cooper [:coop] (he/him)]

There's a list of 16 lion machines in https://bugzilla.mozilla.org/show_bug.cgi?id=759466#c9 that need to be configured in puppet and slavealloc. 6 are destined for preproduction, 8 for the build pool, and 2 for try. Let's break them down as follows:

Build:
bld-lion-r5-081.build.releng.scl3
bld-lion-r5-082.build.releng.scl3
bld-lion-r5-083.build.releng.scl3
bld-lion-r5-084.build.releng.scl3
bld-lion-r5-085.build.releng.scl3
bld-lion-r5-086.build.releng.scl3
bld-lion-r5-087.build.releng.scl3
bld-lion-r5-088.build.releng.scl3

Preproduction:
bld-lion-r5-089.build.releng.scl3
bld-lion-r5-090.build.releng.scl3
bld-lion-r5-091.build.releng.scl3
bld-lion-r5-092.build.releng.scl3
bld-lion-r5-093.build.releng.scl3
bld-lion-r5-094.build.releng.scl3

Try:
bld-lion-r5-095.try.releng.scl3
bld-lion-r5-096.try.releng.scl3

Comment 1

[Amy Rich [:arr] [:arich]]

When you're making buildbot changes, r5-mini-001, r5-mini-002, r5-mini-003, r5-mini-004, r5-mini-005, r5-mini-006 should come out of service (they're being replaced by the above mentioned machines).

Comment 2

[Aki Sasaki (not active)]

Attachment: puppet patch

Comment 3

[Chris Cooper [:coop] (he/him)]

I've added these machines to slavealloc now too.

Comment 4

[Aki Sasaki (not active)]

Comment on attachment 641544 [details] [diff] [review]
puppet patch

http://hg.mozilla.org/build/puppet-manifests/rev/185acbf0b3bb

Comment 5

[Aki Sasaki (not active)]

Attachment: remove r5-mini's per arr

Comment 6

[Aki Sasaki (not active)]

Tried changing the password on 089 and 090 (got this working on 045 earlier; bug 760093 comment 6 ).

This led to me not being able to log in via screen sharing.
Perhaps we should wait til the password is fixed on the image and reimage.

Comment 7

[Chris Cooper [:coop] (he/him)]

Comment on attachment 641612 [details] [diff] [review]
remove r5-mini's per arr

Review of attachment 641612 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/staging_config.py
@@ +1,3 @@
>  import production_config as pc
>  
> +MAC_LION_MINIS = []

r+ if you add the new dev lion minis in their place (already in slavealloc):

MAC_LION_MINIS = ['bld-lion-r5-%03d' % x for x in range(89,95)]

Comment 8

[Chris Cooper [:coop] (he/him)]

cc-ing Kim because she's currently listed in slavealloc as using r5-mini-006 for puppet testing. I don't think that machine is currently plugged in to regular puppet, so as long as IT doesn't take the machine away immediately, she may not care.

Comment 9

[Kim Moir [:kmoir] ET]

Right, I'm just using it for staging testing o the old puppet.

Comment 10

[Aki Sasaki (not active)]

Comment on attachment 641612 [details] [diff] [review]
remove r5-mini's per arr

http://hg.mozilla.org/build/buildbot-configs/rev/3d11f591b264

Comment 11

[Aki Sasaki (not active)]

Ok.

So using Bear's script I was able to fix the password situation, then added the appropriate ssh keys, and puppetized.

Enabling in slavealloc & rebooting brought them back online.

However, scl3-production-puppet is overwriting staging and try ssh keys with production keys.
I don't know how to fix this; until we either decide these are all production boxes or fix this, I'm leaving 89 through 96 disabled in slavealloc.

Comment 12

[Aki Sasaki (not active)]

Attachment: add lion builders to m-r

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 642108 [details] [diff] [review]
add lion builders to m-r

http://hg.mozilla.org/graphs/rev/b658a2015d41 , added via http://graphs1.dmz.scl3.mozilla.com:8080/

Comment 14

[Chris Cooper [:coop] (he/him)]

(In reply to Aki Sasaki [:aki] from comment #11)
> However, scl3-production-puppet is overwriting staging and try ssh keys with
> production keys.
> I don't know how to fix this; until we either decide these are all
> production boxes or fix this, I'm leaving 89 through 96 disabled in
> slavealloc.

I checked the keys on the two try slaves (95,96), and they looked right, even after a reboot, so I've enabled these two slaves. 

Looking at the staging slaves now.

Comment 15

[Chris Cooper [:coop] (he/him)]

I think we're good here now.

Comment 17

[Amy Rich [:arr] [:arich]]

Can we reclaim r5-mini* now?

Comment 18

[Chris Cooper [:coop] (he/him)]

(In reply to Amy Rich [:arich] [:arr] from comment #17)
> Can we reclaim r5-mini* now?

Almost. Filed bug 776888.