921464 - Please create new puppetagain-dash2

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

This host is a clone of puppetagain-dash1.private.scl3.  It's already set up in puppet, inventory, DNS.  It will need to be created, adapter setup in inventory, kickstarted (RHEL), puppetized, and added to nagios.

https://inventory.mozilla.org/en-US/systems/show/10974/

Comment 1

[Rick Bryce [:rbryce]]

VM Kickstarted from template, +puppetize, +RHN, +nagios, +inventory_nic0

Comment 2

[Rick Bryce [:rbryce]]

I didnt watch the output of the puppetize script.  Looks like this node is having issues installing packages via puppet.  Not sure the cause, just noticed it.

Comment 3

[Rick Bryce [:rbryce]]

I rekicked the server and re-puppetized.  I'm getting multiple failure for things like facter, puppet, and puppet_dashboard. Investigating further, hope to have this for you tomorrow.


Warning: /Stage[main]/Puppet_dashboard/Mount[/usr/share/puppet-dashboard/spool]: Skipping because of failed dependencies
Notice: /Stage[main]/Puppet_dashboard::Workers/Service[puppet-dashboard-workers]: Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL] has failures: true
Notice: /Stage[main]/Puppet_dashboard::Workers/Service[puppet-dashboard-workers]: Dependency Group[puppet-dashboard] has failures: true
Warning: /Stage[main]/Puppet_dashboard::Workers/Service[puppet-dashboard-workers]: Skipping because of failed dependencies
Notice: /Stage[main]/Puppet_dashboard/File[/usr/share/puppet-dashboard/config/settings.yml]: Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL] has failures: true
Notice: /Stage[main]/Puppet_dashboard/File[/usr/share/puppet-dashboard/config/settings.yml]: Dependency Group[puppet-dashboard] has failures: true
Warning: /Stage[main]/Puppet_dashboard/File[/usr/share/puppet-dashboard/config/settings.yml]: Skipping because of failed dependencies
Notice: Finished catalog run in 232.26 seconds

Comment 4

[Eric Ziegenhorn :ericz]

This alerted for several things I think because nrpe isn't running due to the above puppet problems.  Downtimed for seven days to get us through the summit.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

It's probably permissions for the NFS mount.  I'll take a peek.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

The initial errors that are causing the dependency failures are

Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: puppet1.private.phx1.mozilla.com]

because it was trying to connect to 'puppet', which resolved to puppet.mozilla.com, which points to phx1.  I have no idea how it got into this state!

Re-puppetizing seems to have helped (after resolving a gid conflict with nagios).  It's up and running and handling reports.  Thanks all!