926466 - Add and enable auditd on puppetagain-hosted linux systems

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

I'll need some help from opsec on how to do this without killing the hosts or changing performance characteristics.  Auditd seems to be buggy and heavyweight, so I'm worried..

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Michael, do you have any particular guidance here?  I don't want to re-invent the wheel, if I don't have to.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

:ulfr, is this something we need to do, or can we just use mig?

Comment 3

[Julien Vehent]

MIG is for investigations. Auditd for monitoring.

MIG does not monitor security events at all. So the only way to keep track of security events on systems is through auditd and logs centralization.

:kang is our local auditd expert.

Comment 4

[Guillaume Destuynder [:kang] [this account is no longer active]]

The current doc is at: https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=26416895

Comment 5

[Guillaume Destuynder [:kang] [this account is no longer active]]

See also configuration in https://github.com/gdestuynder/audisp-json

Comment 6

[Julien Vehent]

Met today with :dustin and :kang.
Notes:
* Documentation is at https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=26416895
* First target is servers. test/build machines should not be in the initial scope. 
* Auditd rules can be ramped up to slowly increase the load while we test.
* Mozdef receives auditd events. CEF events route through syslog. JSON events HTTP POST directly to Mozdef's API endpoint.

action items:
* provide RPM/DEB packages for 32/64bits (:kang)
* create custom repos and auditd module in puppetagain
* short term use auditd-cef to log to local syslog

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

We'd probably start with this on servers and see how it goes.  Deploying it on test machines would come later -- there are the usual concerns with memory (messages are cached to a configurable level) and CPU usage.

For the moment, this is Linux-only (Ubuntu and CentOS).  OS X can do auditd, but that's not planned yet.

It doesn't look like we'd have a lot of trouble with kernel versions - the LKM should work with all of the versions we have.

On Ubuntu, there are different versions of libaudit, so we'd need to install a different compiled version of auditd for each Ubuntu dist.

We'll eventually need 32-bit packages, although all of our servers are 64-bit, so that's a place to start.  These would be deployed in custom apt and yum repos on the puppet masters, and then installed via a puppet package.  The audisp-{cef,json} packages are easy to build with a make command (via fpm).  auditd itself is in the upstream centos/ubuntu repos.

Shipping logs to mozdef: infra does this via the syslog servers (in CEF).  We could also ship logs directly to mozdef (via https with JSON).  We'll want to figure out how much data we'd be slinging around before turning it on (and can test this in relabs).  We can start by sending to the infra syslog servers and go from there.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

We're going to consider the audisp-{cef,json} packages to be upstream from puppetagain, so we won't have a .spec in the puppet manifests -- just like for mig-agent.

So once I have those packages, I'll get started deploying this in relabs, then on production servers.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Apparently the packages are all in mrepo.  Silly me :)

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Note to self: we need to enable a nagios check to make sure auditd is running, too.

Comment 11

[Guillaume Destuynder [:kang] [this account is no longer active]]

If necessary, we also have centos7 packages now :)
I can make more if needed, otherwise: make rpm / make dev automagically make them with fpm.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug926466-centos.patch

This will need to partner with the log aggregation work for it to do anything useful (as it is, the logs just go to the local syslog).

We don't have any Ubuntu servers, so I haven't added Ubuntu support yet, but when we cover slaves we'll need to add that.

Comment 13

[Justin Wood (:Callek)]

Comment on attachment 8501329 [details] [diff] [review]
bug926466-centos.patch

Review of attachment 8501329 [details] [diff] [review]:
-----------------------------------------------------------------

I may not get to this review until early next week, if its needed sooner, please redirect or discuss with me

Comment 14

[:kanban-engops]

A Pivotal Tracker story has been created for this Bug: https://www.pivotaltracker.com/story/show/80405248

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

> I may not get to this review until early next week, if its needed sooner, please redirect > or discuss with me

...3 weeks later...

Comment 16

[Justin Wood (:Callek)]

Comment on attachment 8501329 [details] [diff] [review]
bug926466-centos.patch

Review of attachment 8501329 [details] [diff] [review]:
-----------------------------------------------------------------

ick sorry this sat for so long

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

Oops -- this typo breaks the dependencies:

diff --git a/modules/auditd/manifests/init.pp b/modules/auditd/manifests/init.pp
index c29da4b..b4a0967 100644
--- a/modules/auditd/manifests/init.pp
+++ b/modules/auditd/manifests/init.pp
@@ -1,16 +1,16 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 class auditd($host_type) {
     include packages::auditd
     include packages::audisp_cef
-    $pacakges = [
+    $packages = [
         Class['packages::auditd'],
         Class['packages::audispd_cef'],
     ]
 
     # filter legitimate host types; these are used by the
     # rules template to decide how to configure the host
     case $host_type {
         'slave': { }

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8501329 [details] [diff] [review]
bug926466-centos.patch

http://hg.mozilla.org/build/puppet/rev/8f193d103625

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

bustage fix in comment 17: http://hg.mozilla.org/build/puppet/rev/56740aafb27b
and another bustage fix: https://hg.mozilla.org/build/puppet/rev/a0257ecc6548

(both typos, the first making the second visible)

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

This adds a lot of lines to /var/log/messages, when perhaps it's of most use to opsec through their systems. Do we have the option of supressing it in /var/log/messages ?