872545 - Please create a new Puppet-3.2.0 server in each AZ

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

We'll need new puppetmasters for 3.2.0 support.  Once we're transitioned, we'll be able to shut the others off.

Let's call these releng-puppet1.srv.releng.{use1,usw2}.mozilla.com.

They'll have the same requirements as the existing masters, and can use the same data.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

ping?

Comment 2

[Rail Aliiev [:rail]]

Sorry for the delay. Here is the list of the VMs

puppetmaster-03.srv.releng.use1.mozilla.com 10.134.48.57 
puppetmaster-04.srv.releng.use1.mozilla.com 10.134.49.5  
puppetmaster-05.srv.releng.use1.mozilla.com 10.134.49.145
                                                         
puppetmaster-06.srv.releng.usw2.mozilla.com 10.132.48.212
puppetmaster-07.srv.releng.usw2.mozilla.com 10.132.48.229
puppetmaster-08.srv.releng.usw2.mozilla.com 10.132.48.96 

SSH key auth should work.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I renamed them:

> puppetmaster-03.srv.releng.use1.mozilla.com 10.134.48.57 
releng-puppet1.srv.releng.use1.mozilla.com
https://inventory.mozilla.org/en-US/systems/show/10513/

> puppetmaster-04.srv.releng.use1.mozilla.com 10.134.49.5  
releng-puppet2.srv.releng.use1.mozilla.com
https://inventory.mozilla.org/en-US/systems/show/10514/

> puppetmaster-05.srv.releng.use1.mozilla.com 10.134.49.145
releng-puppet3.srv.releng.use1.mozilla.com
https://inventory.mozilla.org/en-US/systems/show/10515/

> puppetmaster-06.srv.releng.usw2.mozilla.com 10.132.48.212
releng-puppet1.srv.releng.usw2.mozilla.com
https://inventory.mozilla.org/en-US/systems/show/10516/

> puppetmaster-07.srv.releng.usw2.mozilla.com 10.132.48.229
releng-puppet2.srv.releng.usw2.mozilla.com
https://inventory.mozilla.org/en-US/systems/show/10517/

> puppetmaster-08.srv.releng.usw2.mozilla.com 10.132.48.96 
releng-puppet3.srv.releng.usw2.mozilla.com
https://inventory.mozilla.org/en-US/systems/show/10518/

DNS is added via inventory.  Thanks!

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

These do not appear to have the necessary flows open between them, even within a VLAN.  In particular, all hosts (in releng.{use1,usw1} and in scl3/mtv1/scl1, including other puppet masters) need access to the masters on tcp/{8140,443,80}.  The masters should be able to access one another on tcp/22.

Comment 5

[Rail Aliiev [:rail]]

This is fixed now.

<rail> dustin: can you verify puppetmaster netflows?
<dustin> rail: looking good!

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Thanks!  These are all running and part of the moco cluster now, and as far as I can see the flows are fine.  They'll take a little while to get everything sync'd, so we may see a fair bit of cronspam from them for a few hours.  /cc Callek since I know he gets excited about such things

Comment 7

[Rail Aliiev [:rail]]

On IRC we decided to use 4 puppet master only in total (AZ vs region confusion). Let's remove one per region from puppet, then I kill the instances.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug872545.patch

Once this is landed, I'll give everyone an hour or so to get the new config, then shut down the releng-puppet3*

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

So, in an hour or so, go ahead and kill releng-puppet3.*

Comment 10

[Rail Aliiev [:rail]]

All done

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Please also remove them from inventory (and thus DNS)

Comment 12

[Rail Aliiev [:rail]]

Done.