1087013 - Move slaves from staging to production

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris Cooper [:coop] (he/him)]

I don't think we need slaves specifically allocated to staging, especially since preprod went away. We could certainly use that capacity in production.

We already have more people on the team than staging slaves available, and we're not shy about grabbing slaves from production when needed. With the upcoming slave loan tool, it will be trivially easy for releng (or anyone) to loan themselves a slave directly from production.

Let's move the specific slaves listed in staging_config.py into production_config.py

Here's the list:

b-2008-ix-0182
b-2008-ix-0183
b-linux64-hp-0020
b-linux64-hp-0021
b-linux64-hp-0022
b-linux64-hp-0023
b-linux64-hp-0024
bld-lion-r5-043
bld-lion-r5-086
bld-lion-r5-092
ix-mn-w0864-001
ix-mn-w0864-002
ix-mn-w0864-003

Note: I'm not going to change anyone's allocation in slavealloc. In fact I think we should be using slavealloc as the tool to decide where the slaves end up rather than hard-coding values in the configs anyway.

Comment 1

[Chris Cooper [:coop] (he/him)]

Attachment: Move staging slaves to production

This patch simplifies things a little bit and consolidates our slave definitions in the production config. We should be using slavealloc to move slaves between buckets.

I'm not sure what we should do with the ix-mn-w0864. I think we should probably rename them to b-2008-ix, and the corresponding relops work that entails.

Comment 2

[Amy Rich [:arr] [:arich]]

ix-mn-w0864 should not be used in production and is named differently because it's a relops-specific test machine.

Comment 3

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8509128 [details] [diff] [review]
Move staging slaves to production

Review of attachment 8509128 [details] [diff] [review]:
-----------------------------------------------------------------

All fine. I half wonder whether we need the arrays of slaves at all, and just allow any slave to connect, if it passes the correct credentials.

AFAIK, this is the only reason we define them at all, to act as a whitelist of allowed slaves to connect to an environment - is that right? If that is the case, maybe we can ditch the arrays altogether, and rely on the password authentication.

In any case - in the current model, these changes are very good (with the exception that Amy suggested we don't use the ix-mn-w0864-ix machines in production).

Thanks Coop!
Pete

::: mozilla/production_config.py
@@ +1,2 @@
> +MAC_LION_MINIS = ['bld-lion-r5-%03d' % x for x in range(1,16) + range(41,69) + \
> +                                                  range(70,87) + range(88,95)]

looks correct

@@ +4,5 @@
>  LINUX64_IXS    = []
>  WIN32_IXS      = []
>  WIN64_IXS      = []
>  WIN64_REV2     = ['b-2008-ix-%04i' % x for x in range(1,18) + range(65,89) + range(90,159) + range(161,173)] + \
> +                 ['ix-mn-w0864-%03d' % x for x in range(1,3)]

Amy says we shouldn't use these ix machines in prod - not sure why.

@@ -4,5 @@
>  LINUX64_IXS    = []
>  WIN32_IXS      = []
>  WIN64_IXS      = []
>  WIN64_REV2     = ['b-2008-ix-%04i' % x for x in range(1,18) + range(65,89) + range(90,159) + range(161,173)] + \
> -                 ['b-2008-sm-%04d' % x for x in range(33, 65)]

removing seamicro machines - good call

@@ +5,5 @@
>  WIN32_IXS      = []
>  WIN64_IXS      = []
>  WIN64_REV2     = ['b-2008-ix-%04i' % x for x in range(1,18) + range(65,89) + range(90,159) + range(161,173)] + \
> +                 ['ix-mn-w0864-%03d' % x for x in range(1,3)]
> +MOCK_DL120G7   = ['b-linux64-hp-%04d' % x for x in range(20,36)]

looks correct

@@ +35,5 @@
>  TRY_WIN64_IXS  = []
>  TRY_LINUX64_EC2 = ['try-linux64-ec2-%03d' % x for x in range(1, 60) + range(301,340)] + \
>      ['try-linux64-spot-%03d' % x for x in range(1, 200) + range(300,500)] + \
>      ['try-linux64-spot-%d' % x for x in range(1000, 1100)]
> +TRY_WIN64_REV2 = ['b-2008-ix-%04i' % x for x in range(18, 65) + range(173,185)]

looks correct

@@ -36,5 @@
>  TRY_LINUX64_EC2 = ['try-linux64-ec2-%03d' % x for x in range(1, 60) + range(301,340)] + \
>      ['try-linux64-spot-%03d' % x for x in range(1, 200) + range(300,500)] + \
>      ['try-linux64-spot-%d' % x for x in range(1000, 1100)]
> -TRY_WIN64_REV2 = ['b-2008-ix-%04i' % x for x in range(18, 65) + range(173,182) + [184]] + \
> -                 ['b-2008-sm-%04d' % x for x in range(1, 33)]

thanks for removing seamicro

@@ -219,5 @@
>      for list_name in list_names:
>          for host_platform in host_info[list_name]:
>              for host_name in host_info[list_name][host_platform]:
>                  print("%s,%s,%s" % (list_name, host_platform, host_name))
> -

be gone, trailing blank line!

::: mozilla/staging_config.py
@@ +2,5 @@
>  import production_config as pc
>  
> +# Import all of our slave information from production.
> +SLAVES = deepcopy(pc.SLAVES)
> +TRY_SLAVES = deepcopy(pc.TRY_SLAVES)

I think this makes sense, moving forwards - simply listing the slave in TRY_SLAVES and/or SLAVES doesn't mean it will connect, simply that it is allowed to connect, right? In which case I agree better for us to define one pool of allowed hostnames (ideally we'd pull this from slavealloc at some point, rather than statically define in mozilla/production_config.py) and then use it in both the staging and production config. If a given slave is assigned to the correct pool and/or master in slavealloc, it should pull back the correct buildbot.tac to connect accordingly, and connect to the correct environment, so I don't see any real problems with slaves being in both configs. It moves the burden of management to slavealloc, which is indeed what we want. =)

Comment 4

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8509128 [details] [diff] [review]
Move staging slaves to production

Review of attachment 8509128 [details] [diff] [review]:
-----------------------------------------------------------------

Based on Amy's comment, I've moved the ix-mn-w0864 back to the staging config.

https://hg.mozilla.org/build/buildbot-configs/rev/1e059ccc9d2c

Comment 5

[Chris Cooper [:coop] (he/him)]

Landed a fix for the now-failing test:

https://hg.mozilla.org/build/buildbot-configs/rev/60a143c169ff

Comment 6

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/60a143c169ff

Comment 7

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/0734394c4f49

Comment 8

[Chris Cooper [:coop] (he/him)]

Should do this for the test slaves too.

Comment 9

[Chris Cooper [:coop] (he/him)]

Attachment: Define all pandas in production

slavealloc is how we decide where these end up, so no point allocating them again in the configs. Cleans things up nicely.

Comment 10

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8511260 [details] [diff] [review]
Define all pandas in production

Review of attachment 8511260 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/buildbot-configs/rev/750369f7260a

Comment 11

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/750369f7260a