758313 - infra puppet changes cause puppet runs in releng.scl3 to fail

Status: RESOLVED FIXED

(Infrastructure & Operations :: Infrastructure: Other, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

[[e-mail]3 shm]# grep 'server =' /etc/puppet/puppet.conf 
    server = puppetmaster2.private.scl3.mozilla.com
[[e-mail]3 shm]# nc -vz -w 1 puppetmaster2.private.scl3.mozilla.com 8140
nc: connect to puppetmaster2.private.scl3.mozilla.com port 8140 (tcp) timed out: Operation now in progress

I'm not sure how this *should* work.  If this is just a flow request, please add it and change to the ACL component?

I worked around this for the moment with

[[e-mail]3 ssl-master]# puppet agent --test --server puppet1.private.scl3.mozilla.com

Comment 1

[Shyam Mani [:fox2mike]]

Off the top of my head, nothing should be the puppetmaster except puppet1.private.scl3 

The other backends are transparent and should never be directly used as puppetmasters.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Puppet is setting it back:

notice: /Stage[main]/Base/File[puppet-conf]/content: 
--- /etc/puppet/puppet.conf     2012-05-24 11:59:51.000000000 -0700
+++ /tmp/puppet-file20120524-453-5xc26b-0       2012-05-24 12:00:17.000000000 -0700
@@ -1,6 +1,6 @@
 [main]
     # Where we connect to
-    server = puppet1.private.scl3.mozilla.com
+    server = puppetmaster3.private.scl3.mozilla.com
 
     # Where Puppet stores dynamic and growing data.
     # The default value is '/var/puppet'.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

It looks like the puppetmaster is being set to the backend master.  I just saw

notice: /Stage[main]/Releng::Puppetmaster::Ssl/Exec[setup-puppetmaster-certs]/returns: executed successfully
info: /Stage[main]/Releng::Puppetmaster::Ssl/Exec[setup-puppetmaster-certs]: Scheduling refresh of Service[httpd]
notice: /Stage[main]/Base/File[puppet-conf]/content: 
--- /etc/puppet/puppet.conf     2012-05-24 12:00:17.000000000 -0700
+++ /tmp/puppet-file20120524-9328-bc943k-0      2012-05-24 16:57:36.000000000 -0700
@@ -1,6 +1,6 @@
 [main]
     # Where we connect to
-    server = puppetmaster3.private.scl3.mozilla.com
+    server = puppetmaster1.private.scl3.mozilla.com
 
     # Where Puppet stores dynamic and growing data.
     # The default value is '/var/puppet'.

which suggests Apache picked puppetmaster1 this time

Comment 4

[Justin Dow [:jabba]]

On any host that has a "puppetmasterX.xxxxxx..." in the server field, please run "puppetd -t --server=puppet1.private.scl3.mozilla.com". This should fix it and it should stay fixed, since I modified puppet.conf template to use the puppetmaster variable instead of the server variable.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

That didn't seem to work:
> [[e-mail]3 ~]# puppet agent --test --server=puppet1.private.scl3.mozilla.com
...
> notice: /Stage[main]/Base/File[puppet-conf]/content: 
> --- /etc/puppet/puppet.conf     2012-05-24 16:57:36.000000000 -0700
> +++ /tmp/puppet-file20120526-29023-1c9rpf8-0    2012-05-26 08:52:32.000000000 -0700
> @@ -1,6 +1,6 @@
>  [main]
>      # Where we connect to
> -    server = puppetmaster1.private.scl3.mozilla.com
> +    server = puppetmaster3.private.scl3.mozilla.com
>  
>      # Where Puppet stores dynamic and growing data.
>      # The default value is '/var/puppet'.

Comment 6

[Justin Dow [:jabba]]

It looks like this is working now. The host has the correct puppetmaster in /etc/puppet/puppet.conf and puppet runs are succeeding.