1006891 - [Ubuntu] Using nagios-nrpe-plugin on nodes which use "toplevel::server" will install Apache

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Jake Watkins [:dividehex]]

Not sure how this happened but in the case of openstack-testing.relabs.releng.scl3.mozilla.com, apache ended up installed and running.  This should be looked into.

c&p from email with :dustin

> I created a VM from the buildmaster template for testing openstack
> puppet module, puppetized it against toplevel::server and noticed it
> installed apache.  Before I go diving on where that is happening, is
> that suppose to be?

I don't think so!  I only see 'include packages::httpd' in the httpd, mozpool and puppetmaster modules, and I only see 'include httpd' or 'httpd::' (which invoke the httpd module) in the bmm, mozpool, puppetmaster, and talos modules.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I'm not sure what you're seeing..

[root@openstack-testing ~]# grep apache /var/log/syslog 
May  5 16:40:47 openstack-testing puppet-agent[1563]: (/Stage[main]/Smarthost::Setup/File[/etc/aliases]/content) +apache: root
[root@openstack-testing ~]# grep httpd /var/log/syslog 

..and..

[root@openstack-testing ~]# sort /var/lib/puppet/classes.txt   | grep httpd
[root@openstack-testing ~]# sort /var/lib/puppet/classes.txt   | grep apache
[root@openstack-testing ~]# sort /var/lib/puppet/classes.txt   | grep packages
packages::crond
packages::editors
packages::mailx
packages::netcat
packages::nrpe
packages::ntp
packages::ntpdate
packages::openssl
packages::postfix
packages::puppet
packages::screen
packages::setup
packages::strace
packages::sudo
packages::tzdata

..and, after stopping apache..

[root@openstack-testing ~]# puppet agent --test --environment=jwatkins
...
Info: Applying configuration version 'edd3c290bfc9'
Notice: /Stage[packagesetup]/Packages::Setup/Exec[apt-get-update-daily]/returns: executed successfully
Notice: Augeas[resolvconf](provider=augeas): 
--- /etc/resolv.conf    2014-05-05 16:43:48.309811849 -0700
+++ /etc/resolv.conf.augnew     2014-05-06 18:28:27.222225801 -0700
@@ -2,4 +2,4 @@
 #     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
 nameserver 10.26.75.40
 nameserver 10.26.75.41
-search relabs.releng.scl3.mozilla.com releng.scl3.mozilla.com scl3.mozilla.com mozilla.com mozilla.org mozilla.net
+domain relabs.releng.scl3.mozilla.com

Notice: /Stage[main]/Network::Resolv/Augeas[resolvconf]/returns: executed successfully
Warning: Augeas[vmware-clocksource](provider=augeas): Loading failed for one or more files, see debug for /augeas//error output
Notice: Finished catalog run in 7.20 seconds

It didn't try to start Apache.  The resolv.conf thing is a known fight between dhcp and puppet.  The augeas bit is bug 1005389.

Comment 2

[Jake Watkins [:dividehex]]

So.... it looks like installing nagios-nrpe-plugin Recommends nagios3 which depends on nagios3-cgi which depends on libapache2-mod-php5 which depends on... you guessed it; Apache!

There is really no reason to have nagios3 installed here since all we want is the nrpe plugins.
:dustin,
I came across, https://github.com/puppetlabs/puppet/pull/1650 but could we just ensure absent nagios3

[root@openstack-testing log]# apt-cache depends nagios-nrpe-plugin
nagios-nrpe-plugin
  Depends: libc6
  Depends: libssl1.0.0
  Recommends: nagios3
  Conflicts: <nagios-nrpe-doc>
  Conflicts: <nagios-nrpe-doc:i386>
  Conflicts: nagios-nrpe-plugin:i386

[root@openstack-testing log]# apt-cache depends nagios3
nagios3
  Depends: nagios3-core
  Depends: nagios3-cgi
  Suggests: nagios-nrpe-plugin
  Conflicts: nagios3:i386

[root@openstack-testing log]# apt-cache depends nagios3-cgi
nagios3-cgi
  Depends: nagios3-common
  Depends: coreutils
    coreutils:i386
  Depends: ucf
  Depends: apache2-utils
 |Depends: debconf
  Depends: <debconf-2.0>
    cdebconf
    debconf
  Depends: libc6
 |Depends: libgd2-noxpm
  Depends: libgd2-xpm
  Depends: adduser
 |Depends: libapache2-mod-php5


... etc ... etc ...

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, we got Ubuntu'd.  What a silly distro.

Ensuring absent would end up installing, then uninstalling, Apache.  Using --no-install-recommends for nagios-nrpe-plugin would be better.

Comment 4

[Jake Watkins [:dividehex]]

(In reply to Dustin J. Mitchell [:dustin] (PTO until ~5/20) from comment #3)
> --no-install-recommends for nagios-nrpe-plugin would be better.

So looking deeper, pull/1650 wasn't accepted but pull/2082 was.  Unfortunately, it was merged into 3.6.0RC1 and we are still on v3.4.2.  I guess we can hold our breath since it will be out "any day" now.

https://github.com/puppetlabs/puppet/commit/d659254

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Will 3.6.0 fix this entirely, or do we need a patch after that's out?  Bug 999661 will land completely early next week.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So we are on Puppet 3.6 now on most of the systems, right? Is that still a problem? And does it only affect Ubuntu or also CentOS?

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

This is an Ubuntu-only issue.  I don't know if we need to make adjustments to our manifests to fix this -- Jake?

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Tossing this onto Jake's rainy-day list to verify.

Comment 9

[Jake Watkins [:dividehex]]

Attachment: patch1006891.txt

This patch takes advantage of the install_options attribute now available to the apt package provider in puppet (>=3.6.0).  --no-install-recommends will ensure apache doesn't get installed along with nagios-nrpe-server and nagios-nrpe-plugins BUT does nothing to remove apache from toplevel::server systems that have already been "victimized" by there package managers.

Tested in relabs

Comment 10

[Jake Watkins [:dividehex]]

Comment on attachment 8472479 [details] [diff] [review]
patch1006891.txt

Review of attachment 8472479 [details] [diff] [review]:
-----------------------------------------------------------------

remote:   https://hg.mozilla.org/build/puppet/rev/50068db17283
remote:   https://hg.mozilla.org/build/puppet/rev/042a2f867be2

Comment 11

[Jake Watkins [:dividehex]]

Looks like we are still trying to work around the having nagios3 installed.  This probably just needs to be removed. We might want to also add a nagios package manifest with ensure => absent and include it in the toplevel::server just to get nagios3 uninstalled

Wed Aug 20 15:35:24 -0700 2014 /Stage[main]/Nrpe::Service/Service[nagios3] (err): Could not evaluate: Could not find init script or upstart conf file for 'nagios3'
Wed Aug 20 15:35:24 -0700 2014 /Stage[main]/Nrpe::Service/Service[nagios3] (err): Failed to call refresh: Could not find init script or upstart conf file for 'nagios3'
Wed Aug 20 15:35:24 -0700 2014 /Stage[main]/Nrpe::Service/Service[nagios3] (err): Could not find init script or upstart conf file for 'nagios3'

Comment 12

[Jake Watkins [:dividehex]]

Attachment: 1006891-2.patch

Patch removes the nagios3 service and adds a temporary nagios3: ensure => absent to the nrpe package manifest.  Once this has run this propagates through every system this nrpe, we can remove it.

Comment 13

[Justin Wood (:Callek)]

Comment on attachment 8476334 [details] [diff] [review]
1006891-2.patch

Review of attachment 8476334 [details] [diff] [review]:
-----------------------------------------------------------------

not really a fan of ensure => absent in packages/* but since this is temporary and due to that module, sure.

Comment 14

[Jake Watkins [:dividehex]]

Comment on attachment 8476334 [details] [diff] [review]
1006891-2.patch

https://hg.mozilla.org/build/puppet/rev/49a04b3fe031
https://hg.mozilla.org/build/puppet/rev/020459219529

merged production by :callek

Comment 15

[Jake Watkins [:dividehex]]

Attachment: 1006891-3.patch

Removes that nagios3 removal workaround.  Any remnants of nagios3 should be removed by now.

Comment 16

[Jake Watkins [:dividehex]]

Comment on attachment 8488087 [details] [diff] [review]
1006891-3.patch

remote:   https://hg.mozilla.org/build/puppet/rev/75e72cda479e
remote:   https://hg.mozilla.org/build/puppet/rev/c71ff7b67534