712398 - Setup buildbot-master21 as a mac test master

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Armen [:armenzg]]

We're seeing in bug 712244 PB Maximum connection being reached and CPU wio.
There are 2 masters for Linux slaves, 3 masters for Macosx and 2 masters for Windows.
Nevertheless, there are slightly more Windows machines per silo as their jobs take longer.

I am adding almost another 30 Windows slaves and would like to be ready for it.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

ok, buildbot-master21 is up in scl1, with the root password changed, and has been added to nagios and inventory.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

over to releng for puppetization/configuration.

Comment 3

[Armen [:armenzg]]

Thanks Dustin!

Comment 4

[Armen [:armenzg]]

I won't be able to touch it until the week of January 9th when I come back from vacations.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

14:16 < nagios-sjc1> buildbot-master21.build.scl1:MySQL connectivity is ACKNOWLEDGEMENT (CRITICAL): CHECK_NRPE: Socket timeout after 10 seconds.;dustin;not set up yet

So the mysql ACL for this isn't in place yet.  The new plan for such tasks is to file a "Server Ops: ACLs" request for the network change, and a "Server Ops: Database" request for the MySQL change, so they can be done in parallel.  Please do so on the 9th, or earlier if someone else picks this up.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

I missed that I need to modify the number of CPUs *after* building the VM.  Arr spotted it, and has modified the VM, so it will come up with 2 CPUs after it's rebooted.  I'll leave it to you guys to schedule that (since I'm not sure if it's in prod yet).

Comment 7

[Armen [:armenzg]]

I'll use bug 712244 to set it up.

Comment 8

[Armen [:armenzg]]

We'll do the setup in here and leave bug 712244 to determine what is the way forward.

Comment 9

[Chris Cooper [:coop] (he/him)]

Armen: you switched from "I'll" in comment #7 to "We'll" in comment #8 -- who is actually going to do the work here?

Comment 10

[Armen [:armenzg]]

I have not had time to work on this for the last month and not seeing the end of the tunnel. Putting back in the queue. This is not urgent. It might be more pressing when there are less r4 machines out of the pool due to dongles.

We now have 3 masters of each testing OS group.
I believe we should have one more macosx test master due to ratio of #slaves/#masters

darwin10/darwin11 -> 170
darwin9 -> 61
win7 -> 73
xp -> 67
fed32 -> 73
fed64 -> 68

tests1-linux -> 73+68=141
tests1-windows -> 73+67=140
tests1-macosx -> 170+61=231

Comment 11

[[e-mail] (:bhearsum)]

Armen, is this bug simply about getting a master instance up and running on this machine, and putting it in the production pool?

Comment 12

[Armen [:armenzg]]

yes, that is correct.

Based on comment 10, having an extra macosx master would be the best use of it.


Thanks!

Comment 13

[[e-mail] (:bhearsum)]

OK, thanks. I'll try to get this done this week.

Comment 14

[[e-mail] (:bhearsum)]

Attachment: update json

Comment 15

[[e-mail] (:bhearsum)]

Attachment: puppet config update

Comment 16

[[e-mail] (:bhearsum)]

I added this master to slavealloc, put the proper ssh keys on it. At this point, I'm just waiting for reviews, then I can turn it on I think....Once it's actually up and running we need to update Nagios to look for it, as this check is currently looking for 0 instances of buildbot ;) https://nagios.mozilla.org/nagios/cgi-bin/extinfo.cgi?type=2&host=buildbot-master21.build.scl1&service=buildbot

Comment 17

[[e-mail] (:bhearsum)]

Comment on attachment 602916 [details] [diff] [review]
puppet config update

This is landed, and the master has been created and turned on. I've pushed talos-r3-leopard-032 and talos-r4-{snow,lion}-032 to it, and after I see successful jobs from them, I'll enable the master fully.

Comment 18

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #16)
> Once
> it's actually up and running we need to update Nagios to look for it, as
> this check is currently looking for 0 instances of buildbot ;)
> https://nagios.mozilla.org/nagios/cgi-bin/extinfo.cgi?type=2&host=buildbot-
> master21.build.scl1&service=buildbot

Apparently Puppet changes this somewhere, so it's OK now.

Comment 19

[[e-mail] (:bhearsum)]

I've seen successful builds from all 3 slaves, I just unlocked them and enabled the master fully. The slaves should rebalance themselves in the next couple of hours.

Comment 20

[[e-mail] (:bhearsum)]

So, nothing else to do here, woo!