744882 - Configure new buildbot masters in scl3

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Chris Cooper [:coop] (he/him)]

As the move from sjc -> scl3 continues, we will need to create new masters in scl3 (or scl1) to replace the ones we currently have in sjc, and point our slave pool at them. This is particularly pressing for releases because we typically use bm07 and bm08 for release work, both of which are in sjc.

slavealloc tells me the following non-dev/pp masters are in sjc currently:

buildbot-master07.build.sjc1.mozilla.com:8001 (build-sjc1)	
buildbot-master08.build.sjc1.mozilla.com:8001 (build-sjc1)
buildbot-master09.build.sjc1.mozilla.com:8101 (try-sjc1)
buildbot-master26.build.sjc1.mozilla.com:8001 (build-sjc1)
buildbot-master27.build.sjc1.mozilla.com:8101 (try-sjc1)

All of these masters have more than one platform of slave connected.

We already have bm30 and bm31 setup in scl3 for lion machines, and bm32 and bug33 are created but not configured yet. We'll need relops to create 3 more masters for us in scl3 (bug 744877), then we'll need to configure those 5 new VMs.

Comment 1

[Chris Cooper [:coop] (he/him)]

Hal: thanks for taking this on.

There are 5 buildbot masters that need to be setup in scl3: 

buildbot-master32.srv.releng.scl3.mozilla.com
buildbot-master33
buildbot-master34
buildbot-master35
buildbot-master36

We should match the build/try distribution from the above comment, i.e. 3 builds masters and 2 try masters. Note: we won't need all the masters immediately. It will suffice to only enable one build master and one try master for the new linux VMs, but we should setup all the masters now so we're ready to point other slaves at them (scl1,mtv) when we turn off the sjc masters.

This will require some puppet updates:

https://hg.mozilla.org/build/puppet-manifests/file/default/buildmaster-production.pp

...and to our tools for managing masters (be sure to only mark as "enabled" the ones that will be used immediately):

https://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/production-masters.json

Comment 2

[Chris Cooper [:coop] (he/him)]

These masters will also need to be added to slavealloc.

Comment 3

[hwine]

Attachment: add masters to production list

All masters are being added with "enabled=false":
$ grep enabled bz744882-01.diff 
+    "enabled": false,
+    "enabled": false,
+    "enabled": false,
+    "enabled": false,
+    "enabled": false,

This needs to land before puppet configs can be verified. After everything is up, there will be a separate patch/land to enable the 2 masters we want on initially

Comment 4

[hwine]

Comment on attachment 616241 [details] [diff] [review]
add masters to production list

http://hg.mozilla.org/build/puppet-manifests/rev/72cc67f01dd1

Comment 5

[hwine]

Comment on attachment 616241 [details] [diff] [review]
add masters to production list

different diff committed to different repository :( -- will back that out first

Comment 6

[hwine]

Comment on attachment 616241 [details] [diff] [review]
add masters to production list

http://hg.mozilla.org/build/tools/rev/de124c4ee91a

Comment 7

[hwine]

Attachment: add masters to puppet production list

Adds masters to puppet configuration, required before puppet daemon can be enabled.

Comment 8

[hwine]

bm32 now does puppet sync okay against my (hwine) environment. With above commit, will be able to enable daemon.

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 616350 [details] [diff] [review]
add masters to puppet production list

Review of attachment 616350 [details] [diff] [review]:
-----------------------------------------------------------------

These will need to get added to slavealloc, too.

Comment 10

[hwine]

Comment on attachment 616350 [details] [diff] [review]
add masters to puppet production list

http://hg.mozilla.org/build/puppet-manifests/rev/40a0d7ed0588

Comment 11

[hwine]

hosts buildbot-master32 & buildbot-master33 configured with active puppet. Will return to rest once slaves online (bug 744067)

Comment 12

[hwine]

Attachment: database additions

slavealloc additions for these new build masters - only bm32 & bm33 are being allocated for now, the others will remain disabled in both slavealloc and production-masters.json

Comment 13

[hwine]

buildmasters added to slavealloc, and disabled

Comment 14

[hwine]

Attachment: enable bm32 & bm33

Time to bring these two online

Comment 15

[hwine]

Comment on attachment 619990 [details] [diff] [review]
enable bm32 & bm33

http://hg.mozilla.org/build/tools/rev/de8c4597982d

Comment 16

[Rail Aliiev [:rail]]

Comment on attachment 619990 [details] [diff] [review]
enable bm32 & bm33

I backed out this change http://hg.mozilla.org/build/tools/rev/87e0fec249f7

the masters aren't ready to be used, at least .ssh is empty, what would prevent them upload log files.

Comment 17

[hwine]

Nice catch - thanks. Should be resolved now.

Comment 18

[hwine]

Comment on attachment 619990 [details] [diff] [review]
enable bm32 & bm33

back in after expanded testing on masters:
http://hg.mozilla.org/build/tools/rev/ea88e614e6f4

Comment 19

[hwine]

status update: the 2 "needed now" masters, bm32 & bm33, are in production, and bm36 is also in production to control self serve.

bm34 & bm35 just need to be puppetized. As there is no urgent need for that, I'm hoping to update the docs, and then bring those up base on the new docs to verify.

If anyone needs them just grab them and let me know in this ticket.

Comment 20

[hwine]

status update: bm34 & bm35 have been puppetized. They were already put into slavealloc and production-masters.json as build & try (respectively). If no changes needed, just enable them in slavealloc

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

You'd need to flip bm34/35 to enabled in production-masters.json too (so they show up to fabric etc).