1036509 - Put 11 new mtnlion test machines into production

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Chris Cooper [:coop] (he/him)]

11 builders were re-imaged as testers in bug 1034715. Let's get them into production.

Comment 1

[Chris Cooper [:coop] (he/him)]

The machines are:

talos-mtnlion-r5-090.test.releng.scl3.mozilla.com
talos-mtnlion-r5-091.test.releng.scl3.mozilla.com
talos-mtnlion-r5-092.test.releng.scl3.mozilla.com
talos-mtnlion-r5-093.test.releng.scl3.mozilla.com
talos-mtnlion-r5-094.test.releng.scl3.mozilla.com
talos-mtnlion-r5-095.test.releng.scl3.mozilla.com
talos-mtnlion-r5-096.test.releng.scl3.mozilla.com
talos-mtnlion-r5-097.test.releng.scl3.mozilla.com
talos-mtnlion-r5-098.test.releng.scl3.mozilla.com
talos-mtnlion-r5-099.test.releng.scl3.mozilla.com
talos-mtnlion-r5-100.test.releng.scl3.mozilla.com

Comment 2

[Chris Cooper [:coop] (he/him)]

Attachment: Add 11 mtnlion slaves to test pool

Comment 3

[Chris Cooper [:coop] (he/him)]

Attachment: Add mtnlion slaves to graph server

These are missing completely from the default data list right now, but have already been added in staging and production. Not awesome, I know, but we should get them back in sync.

Comment 4

[Jordan Lund (:jlund)]

Comment on attachment 8453245 [details] [diff] [review]
Add mtnlion slaves to graph server

Review of attachment 8453245 [details] [diff] [review]:
-----------------------------------------------------------------

I think we want "talos-mtnlion-r5-XXX" here no?

I assume we either didn't put these in staging/production or they would have blown up as dups?

Comment 5

[Jordan Lund (:jlund)]

Comment on attachment 8453236 [details] [diff] [review]
Add 11 mtnlion slaves to test pool

Review of attachment 8453236 [details] [diff] [review]:
-----------------------------------------------------------------

just a sanity check Bug 1034715 specifies 10 machines: 90-99. this bug here specifies 11: 90-100

I'm guessing we just had another machine that buildbot didn't know about but something also seems wrong with hostname resolve (look at 090):
> ssh cltbld@talos-mtnlion-r5-090
[[e-mail] ~]$ exit

> ssh cltbld@talos-mtnlion-r5-091
[[e-mail] ~]$ exit

> ssh cltbld@talos-mtnlion-r5-100
[[e-mail] ~]$

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to Jordan Lund (:jlund) from comment #5) 
> just a sanity check Bug 1034715 specifies 10 machines: 90-99. this bug here
> specifies 11: 90-100
> 
> I'm guessing we just had another machine that buildbot didn't know about but
> something also seems wrong with hostname resolve (look at 090)

Yes, Amy returned bld-lion-r5-087 on loan in bug 992378 as a mtnlion tester at my request.

Comment 7

[Chris Cooper [:coop] (he/him)]

Attachment: Add mtnlion slaves to graph server, v2

(In reply to Jordan Lund (:jlund) from comment #4)
> I think we want "talos-mtnlion-r5-XXX" here no?
> 
> I assume we either didn't put these in staging/production or they would have
> blown up as dups?

Yes, sorry. Cut-n-paste fail here on my part.

These aren't getting added to production/stage because they're already there. I'll just trying to back-fill the graphsserver bootstrapping sql.

Comment 8

[Jordan Lund (:jlund)]

Comment on attachment 8453236 [details] [diff] [review]
Add 11 mtnlion slaves to test pool

Review of attachment 8453236 [details] [diff] [review]:
-----------------------------------------------------------------

Ah OK. And I assume you are confident that it won't be an issue if talos-mtnlion-r5-090 thinks its host name is different than its fqdn as you didn't touch on that part of the review comment. My primitive understanding thought maybe this machine was imaged incorrectly or it would have puppetizing/buildbot issues.

sorry to comment on this again but looking at this slave, my spidey sense says this slave was not re-imaged and may warrant confirmation of amy or whoever did it.

jlund@Hastings163:~BUDM (*)
> ssh cltbld@talos-mtnlion-r5-090                                                                                              build-cfg-integration [47aabad] modified 
[[e-mail] ~]$ ls /builds/slave/
buildbot.tac                    try-osx64-00000000000000000000  twistd.log                      twistd.log.3                    twistd.log.7
reboot_count.txt                try-osx64-d-000000000000000000  twistd.log.1                    twistd.log.4                    twistd.log.8
tb-try-c-cen-osx64-00000000000  try-osx64_g-000000000000000000  twistd.log.10                   twistd.log.5                    twistd.log.9
tb-try-c-cen-osx64-d-000000000  twistd.hostname                 twistd.log.2                    twistd.log.6

tail /var/log/system.log
Jul  9 21:39:04 bld-lion-r5-038 com.apple.launchd[1] (org.collectd.collectd): Throttling respawn: Will start in 10 seconds
Jul  9 21:39:14 bld-lion-r5-038 collectd[32465]: Looking up "bld-lion-r5-038.try.releng.scl3.mozilla.com" failed. You have set the "FQDNLookup" option, but I cannot resolve my ho
stname to a fully qualified domain name. Please fix you network configuration.
Jul  9 21:39:14 bld-lion-r5-038 com.apple.launchd[1] (org.collectd.collectd[32465]): Exited with code: 1
Jul  9 21:39:14 bld-lion-r5-038 com.apple.launchd[1] (org.collectd.collectd): Throttling respawn: Will start in 10 seconds
Jul  9 21:39:24 bld-lion-r5-038 collectd[32478]: Looking up "bld-lion-r5-038.try.releng.scl3.mozilla.com" failed. You have set the "FQDNLookup" option, but I cannot resolve my 

/var/log/system.log.7.bz2 goes back to july 1st.
 

Either way r+ for what this patch is trying to achieve.

Comment 9

[Chris Cooper [:coop] (he/him)]

(In reply to Jordan Lund (:jlund) from comment #8)
> Ah OK. And I assume you are confident that it won't be an issue if
> talos-mtnlion-r5-090 thinks its host name is different than its fqdn as you
> didn't touch on that part of the review comment. My primitive understanding
> thought maybe this machine was imaged incorrectly or it would have
> puppetizing/buildbot issues.

The fact that it's still displaying it's old hostname means something is amiss here. Maybe it just got missed in the re-imaging? 

Since the old version of this machine (bld-lion-r5-038) is disabled in slavealloc, it doesn't really matter for now. I've kicked off a re-image for this slave using the test cluster commands from the mana page to see if that fixes it. I'm obviously not going to enable the other slaves now. I'll do that and check the result of the re-image in the morning.

Comment 10

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8453236 [details] [diff] [review]
Add 11 mtnlion slaves to test pool

Review of attachment 8453236 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/buildbot-configs/rev/4695ddab8666

Comment 11

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8453457 [details] [diff] [review]
Add mtnlion slaves to graph server, v2

Review of attachment 8453457 [details] [diff] [review]:
-----------------------------------------------------------------

http://hg.mozilla.org/graphs/rev/f3b5552dba34

Comment 12

[Chris Cooper [:coop] (he/him)]

Attachment: Move 3 preprod lion build slaves into production

Comment 13

[Jordan Lund (:jlund)]

Comment on attachment 8453953 [details] [diff] [review]
Move 3 preprod lion build slaves into production

Review of attachment 8453953 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm with one in line check.

::: mozilla/production_config.py
@@ +1,1 @@
> +MAC_LION_MINIS = ['bld-lion-r5-%03d' % x for x in range(1,16) + range(41,69) + range(70,87) + range(89,92) + range(93,95)]

sanity check, do we want to remove bld-lion-r5-088? I still see it in slavealloc and says it's enabled.

Comment 14

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8453953 [details] [diff] [review]
Move 3 preprod lion build slaves into production

Review of attachment 8453953 [details] [diff] [review]:
-----------------------------------------------------------------

Got an r+ from jlund in IRC, provided I add back in 088.

https://hg.mozilla.org/build/buildbot-configs/rev/d9062d7fa837

Comment 15

[Chris Cooper [:coop] (he/him)]

Merged to production, and deployed.

Comment 16

[Chris Cooper [:coop] (he/him)]

I've enabled these slave in slavealloc, and have rebooted them. They should begin taking jobs shortly.

Comment 17

[Chris Cooper [:coop] (he/him)]

These are all taking jobs now.

Take that, wait times!