971812 - create and puppetize 10 additional buildbot masters in scl3

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Justin Wood (:Callek)]

+++ This bug was initially created as a clone of Bug #940659 +++

We need to stand up 10 additional buildbot masters in scl3 on vmware.

buildbot-master103.srv.releng.scl3.mozilla.com
buildbot-master1{04..11}.srv.releng.scl3.mozilla.com
buildbot-master112.srv.releng.scl3.mozilla.com

4G RAM, 40G HD, 4GB swap(?), 2 vCPU
(aka: identical to buildbot-master102.srv.releng.scl3.mozilla.com)

Comment 1

[Amy Rich [:arr] [:arich]]

This will take roughly half a day to complete. It uses the same resources as the bare-metal provisioning project (which is already two weeks behind due to the previous network issues in scl3) and the buildapi cutover (which is already agreed to for the next window). Can you please state where this falls in the priority and which work should be displaced?

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to Amy Rich [:arich] [:arr] from comment #1)
> This will take roughly half a day to complete. It uses the same resources as
> the bare-metal provisioning project (which is already two weeks behind due
> to the previous network issues in scl3) and the buildapi cutover (which is
> already agreed to for the next window). Can you please state where this
> falls in the priority and which work should be displaced?

I would put it behind buildapi (already agreed to) and ahead of bare-metal provisioning. In-house test masters are something that we *know* will improve some network-related failure cases.

Comment 3

[(dormant account)]

(In reply to Chris Cooper [:coop] from comment #2)
> I would put it behind buildapi (already agreed to) and ahead of bare-metal
> provisioning. In-house test masters are something that we *know* will
> improve some network-related failure cases.

r+

Comment 4

[Justin Wood (:Callek)]

dustin poked me in IRC for landing: https://hg.mozilla.org/build/puppet/rev/e962b79ef51e

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

All of these bug 103 are ready.  They all had various vmware bugs, but 103's bugs are particularly intransigent.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

103's ready now, too.  I've added these to Nagios and created CNAMEs.

For the record, total elapsed time: 3:30

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Still needs flows..

Comment 8

[Justin Wood (:Callek)]

Lets assume all flows are in place with resolved Bug 972508 -- I'll throw in a new netops bug if that proves to be an invalid assumption.

Comment 9

[Justin Wood (:Callek)]

:dustin, somethings wrong with the CNAMEs

Justin@AQUARIUS ~
$ ping buildbot-master103.build.mozilla.org
Ping request could not find host buildbot-master103.build.mozilla.org. Please check the name and try again.

Justin@AQUARIUS ~
$ ping buildbot-master103.srv.releng.scl3.mozilla.com

Pinging buildbot-master103.srv.releng.scl3.mozilla.com [10.26.48.26] with 32 bytes of data:
Reply from 10.26.48.26: bytes=32 time=100ms TTL=60
Reply from 10.26.48.26: bytes=32 time=96ms TTL=60

Ping statistics for 10.26.48.26:
    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 96ms, Maximum = 100ms, Average = 98ms
Control-C

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Huh, somehow they weren't either public or private.  I've fixed that and it should propagate soon.