1049956 - Puppet catalog compilation on nagios1.private.releng.scl3.mozilla.com is CRITICAL: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parse template nagios/nagios-host.cfg.erb:

Reporter

Description

•

10 years ago

Automated alert report from nagios1.private.releng.scl3.mozilla.com:

Hostname: nagios1.private.releng.scl3.mozilla.com
Service:  Puppet catalog compilation
State:    CRITICAL
Output:   Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parse template nagios/nagios-host.cfg.erb:

Runbook:  http://m.allizom.org/Puppet+catalog+compilation

david garvey:dgarvey

Comment 1

•

10 years ago

[root@nagios1.private.releng.scl3 ~]# puppetctl run
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/puppetdb_keystore_pw.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pythonversion.rb
Info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ldapvip.rb
Info: Loading facts in /var/lib/puppet/lib/facter/mysql_bytes_sent.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppetmaster.rb
Info: Loading facts in /var/lib/puppet/lib/facter/intel_intremap_bug.rb
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/raidcontroller.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/rhnproxy.rb
Info: Loading facts in /var/lib/puppet/lib/facter/openssl.rb
Info: Loading facts in /var/lib/puppet/lib/facter/env_http_proxy.rb
Info: Loading facts in /var/lib/puppet/lib/facter/postgres_tps.rb
Info: Loading facts in /var/lib/puppet/lib/facter/physical_memory_mb.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ipmi.rb
Info: Loading facts in /var/lib/puppet/lib/facter/customFacts.rb
Info: Loading facts in /var/lib/puppet/lib/facter/mysql_database_size.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/lldp.rb
Info: Loading facts in /var/lib/puppet/lib/facter/mysql_server_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/openssl097a.rb
Info: Loading facts in /var/lib/puppet/lib/facter/vmware_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/partitions.rb
Info: Loading facts in /var/lib/puppet/lib/facter/available_package_updates.rb
Info: Loading facts in /var/lib/puppet/lib/facter/archlib.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hp_server_info.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hp_oa_info.rb
Info: Loading facts in /var/lib/puppet/lib/facter/openssl098e.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hp_storage_firmware.rb
Info: Loading facts in /var/lib/puppet/lib/facter/default_gateway.rb
Info: Loading facts in /var/lib/puppet/lib/facter/mysql_myisam_table_count.rb
Info: Loading facts in /var/lib/puppet/lib/facter/mozec2.rb
Info: Loading facts in /var/lib/puppet/lib/facter/mysql_databases.rb
Info: Loading facts in /var/lib/puppet/lib/facter/sku_number.rb
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hp_array_parser.rb
Info: Loading facts in /var/lib/puppet/lib/facter/mysql_bytes_recd.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hp_oob_ip.rb
Info: Loading facts in /var/lib/puppet/lib/facter/needs_reboot_for_intremap.rb
Info: Loading facts in /var/lib/puppet/lib/facter/rsyslogserver.rb
Info: Loading facts in /var/lib/puppet/lib/facter/mysql_client_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/datacenter.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hp_ethernet_firmware.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ec2.rb
Info: Loading facts in /var/lib/puppet/lib/facter/postgres_server_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/postgres_databases.rb
Info: Loading facts in /var/lib/puppet/lib/facter/nr_ruby_found.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hp_ilo_versions.rb
Info: Loading facts in /var/lib/puppet/lib/facter/mysql_non_utf8_databases.rb
Info: Loading facts in /var/lib/puppet/lib/facter/postgres_database_size.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hp_lo100_info.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hp_pmc_versions.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hp_ilo_lic.rb
Info: Loading facts in /var/lib/puppet/lib/facter/mysql_qps.rb
Info: Loading facts in /var/lib/puppet/lib/facter/needs_reboot_for_selinux.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hp_if_info.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hp_bios.rb
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parse template nagios/nagios-host.cfg.erb:
  Filepath: /usr/lib/ruby/1.8/resolv.rb
  Line: 93
  Detail: no address for mac-v2-signing2.srv.releng.scl3.mozilla.com
 at /etc/puppet/modules/nagios/manifests/releng/scl3.pp:21072 on node nagios1.private.releng.scl3.mozilla.com
Error: Cached catalog for nagios1.private.releng.scl3.mozilla.com failed: Could not parse YAML data for catalog nagios1.private.releng.scl3.mozilla.com: allocator undefined for Proc
Notice: Using cached catalog
Error: Could not retrieve catalog; skipping run
[root@nagios1.private.releng.scl3 ~]#

david garvey:dgarvey

Comment 2

•

10 years ago

[root@nagios1.private.releng.scl3 ~]# ping mac-v2-signing2.srv.releng.scl3.mozilla.com
ping: unknown host mac-v2-signing2.srv.releng.scl3.mozilla.com
[root@nagios1.private.releng.scl3 ~]# 

Not in puppet svn.
dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios/manifests/releng$ grep mac-v2-signing2.srv.releng scl3.pp 
dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios/manifests/releng$

david garvey:dgarvey

Comment 3

•

10 years ago

I was thinking that it would use releng nagios puppet config? But after checking the puppetmaster for releng nagios it is pointing to puppet1.private.scl3.


[root@puppet1.private.scl3 ~]# ls /etc/puppet/modules/nagios/manifests/releng/scl3.pp
/etc/puppet/modules/nagios/manifests/releng/scl3.pp
[root@puppet1.private.scl3 ~]# grep mac-v2-signing2.srv.releng /etc/puppet/modules/nagios/manifests/releng/scl3.pp
        'mac-v2-signing2.srv.releng.scl3.mozilla.com' => {
[root@puppet1.private.scl3 ~]#

david garvey:dgarvey

Comment 4

•

10 years ago

More info:
[root@nagios1.private.releng.scl3 ~]# grep server /etc/puppet/puppet.conf 
    server = puppet1.private.scl3.mozilla.com
    report_server = puppet1.private.scl3.mozilla.com
    # or posting reports to the puppetmaster. Some of our servers
[root@nagios1.private.releng.scl3 ~]#

david garvey:dgarvey

Comment 5

•

10 years ago

dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios$ svn diff -r PREV:HEAD manifests/releng/scl3.pp
Index: manifests/releng/scl3.pp
===================================================================
--- manifests/releng/scl3.pp	(revision 91588)
+++ manifests/releng/scl3.pp	(revision 91641)
@@ -1750,12 +1750,18 @@
                 'pxe-servers'
             ]
         },
-        'mac-signing1.srv.releng.scl3.mozilla.com' => {
+        'mac-v2-signing1.srv.releng.scl3.mozilla.com' => {
             contact_groups => 'build',
             hostgroups => [
                 'mac-signing-servers'
             ]
         },
+        'mac-v2-signing2.srv.releng.scl3.mozilla.com' => {
+            contact_groups => 'build',
+            hostgroups => [
+                'mac-signing-servers'
+            ]     
+        },    
         'mac-signing2.srv.releng.scl3.mozilla.com' => {
             contact_groups => 'build',
             hostgroups => [
dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios$

david garvey:dgarvey

Comment 6

•

10 years ago

dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios$ svn info  manifests/releng/scl3.pp 
Path: manifests/releng/scl3.pp
Name: scl3.pp
Working Copy Root Path: /home/dgarvey/svn/sysadmins
URL: svn+ssh://svn.mozilla.org/sysadmins/puppet/trunk/modules/nagios/manifests/releng/scl3.pp
Relative URL: ^/puppet/trunk/modules/nagios/manifests/releng/scl3.pp
Repository Root: svn+ssh://svn.mozilla.org/sysadmins
Repository UUID: 65d70b31-241a-0410-a745-f8fa14660ae9
Revision: 91641
Node Kind: file
Schedule: normal
Last Changed Author: arich@mozilla.com
Last Changed Rev: 91589
Last Changed Date: 2014-08-06 10:03:44 -0700 (Wed, 06 Aug 2014)
Text Last Updated: 2014-08-06 18:33:58 -0700 (Wed, 06 Aug 2014)
Checksum: bdf68e418aebf35987a896e270b008e0468e5fce

dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios$

david garvey:dgarvey

Comment 7

•

10 years ago

1 is fine but 2 doesn't exist in the network.

[root@puppet1.private.scl3 puppet]# ping mac-v2-signing1.srv.releng.scl3.mozilla.com
PING mac-v2-signing1.srv.releng.scl3.mozilla.com (10.26.48.33) 56(84) bytes of data.
64 bytes from mac-v2-signing1.srv.releng.scl3.mozilla.com (10.26.48.33): icmp_seq=1 ttl=61 time=19.0 ms
64 bytes from mac-v2-signing1.srv.releng.scl3.mozilla.com (10.26.48.33): icmp_seq=2 ttl=61 time=15.2 ms
64 bytes from mac-v2-signing1.srv.releng.scl3.mozilla.com (10.26.48.33): icmp_seq=3 ttl=61 time=20.4 ms
^C
--- mac-v2-signing1.srv.releng.scl3.mozilla.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2173ms
rtt min/avg/max/mdev = 15.280/18.242/20.434/2.176 ms
[root@puppet1.private.scl3 puppet]# 


+        'mac-v2-signing2.srv.releng.scl3.mozilla.com' => {
+            contact_groups => 'build',
+            hostgroups => [
+                'mac-signing-servers'
+            ]     
+        },

Nick Thomas [:nthomas] (UTC+12)

Comment 8

•

10 years ago

Fallout from bug 1049546 I think, there's no A record pointing mac-v2-signing2.srv.releng.scl3.mozilla.com to 10.26.49.25. Inventory has that marked as an OOB IP on a spare mini at the moment.

david garvey:dgarvey

Comment 9

•

10 years ago

Commented this out.


+        'mac-v2-signing2.srv.releng.scl3.mozilla.com' => {
+            contact_groups => 'build',
+            hostgroups => [
+                'mac-signing-servers'
+            ]     
+        },


dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios/manifests/releng$ svn commit -m "couple of bugs here commected mac-v2-signing2.srv.releng.scl3.mozilla.com bugs: 1049956, 1049546"
Sending        scl3.pp
Transmitting file data .
Committed revision 91642.
dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios/manifests/releng$

Dustin J. Mitchell [:dustin] (he/him)

Comment 10

•

10 years ago

Thanks -- I'll clean it up tomorrow after all three of the new hosts are up.

david garvey:dgarvey

Comment 11

•

10 years ago

Yet another host not in inventory or pingable: t-snow-r4-0166.test.releng.scl3.mozilla.com


dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios$ svn diff -r PREV:HEAD manifests/releng/scl3.pp
Index: manifests/releng/scl3.pp
===================================================================
--- manifests/releng/scl3.pp	(revision 91643)
+++ manifests/releng/scl3.pp	(revision 91644)
@@ -8189,14 +8189,15 @@
                 't-snow-r4-machines'
             ]
         },
-        't-snow-r4-0166.test.releng.scl3.mozilla.com' => {
-            contact_groups => 'nobody',
-            parents => 'switch1.r201-8.ops.releng.scl3.mozilla.net',
-            servicegroups => 'scl3-t-snow-r4-ping',
-            hostgroups => [
-                't-snow-r4-machines'
-            ]
-        },
+# Bug 1049956
+#        't-snow-r4-0166.test.releng.scl3.mozilla.com' => {
+#            contact_groups => 'nobody',
+#            parents => 'switch1.r201-8.ops.releng.scl3.mozilla.net',
+#            servicegroups => 'scl3-t-snow-r4-ping',
+#            hostgroups => [
+#                't-snow-r4-machines'
+#            ]
+#        },
         't-snow-r4-0002.test.releng.scl3.mozilla.com' => {
             contact_groups => 'nobody',
             servicegroups => 'scl3-t-snow-r4-ping',
dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios$

Dustin J. Mitchell [:dustin] (he/him)

Comment 12

•

10 years ago

Ugh, sorry again.  Those should be the only two, anyway.  I forgot nagios used DNS -- the host hasn't moved yet :/

MOC Nagios API

Reporter

Comment 13

•

10 years ago

Automated alert recovery:

Hostname: nagios1.private.releng.scl3.mozilla.com
Service:  Puppet catalog compilation
State:    OK
Output:   Puppet agent 3.2.4 running catalog 91643

Status: NEW → RESOLVED

Closed: 10 years ago

Resolution: --- → FIXED

Nobody; OK to take it and work on it

Assignee

Updated

•

9 years ago

Product: mozilla.org → mozilla.org Graveyard

Bugzilla

Quick Search

Puppet catalog compilation on nagios1.private.releng.scl3.mozilla.com is CRITICAL: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parse template nagios/nagios-host.cfg.erb:

Categories

(mozilla.org Graveyard :: Server Operations: MOC, task)

Tracking

(Not tracked)

People

(Reporter: nagiosapi, Unassigned)

References

(
URL
)

Details

(Whiteboard: [id=nagios1.private.releng.scl3.mozilla.com:11141174])

Crash Data

Security

(public)

User Story

Description

Comment 1

Comment 2

Comment 3

Comment 4

Comment 5

Comment 6

Comment 7

Comment 8

Comment 9

Comment 10

Comment 11

Comment 12

Comment 13

Updated