729776 - mtv releng ganeti cluster nodes spewing errors

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Amy Rich [:arr] [:arich]]

After the memory replacement for kvm1.build.mtv1.mozilla.com, I took a look at /var/log/messages and noticed that the machine was spewing tens of thousands of errors of the form:

Feb 21 14:49:46 kvm1 kernel: audit: error converting sid to string

I thought this might be fallout from the RAM replacement, but a took a look at the other kvm nodes in this cluster and they were all spewing the same error.  The kvm nodes in scl1 (not RHEL and not puppetized unlike the ones in mtv) are not exhibiting this behavior.

For the time being I've migrated all of the vms off of kvm1.build.mtv1.mozilla.com so it can be used to help debug the situation.

Comment 1

[Dumitru Gherman [:dumitru]]

Nothing to worry about.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

This is a known problem with auditd, which is one of infrasec's tools.  Almost all of our systems spew this error.  It can be fixed with a kernel upgrade.  I don't know how you feel about the risk/reward for this particular host, though.

Comment 3

[Amy Rich [:arr] [:arich]]

Has serverops tested the kernel upgrade on any machines yet?  How about any ganeti machines?

Comment 4

[Dumitru Gherman [:dumitru]]

(In reply to Amy Rich [:arich] [:arr] from comment #3)
> Has serverops tested the kernel upgrade on any machines yet?  
Yes, and it's fine.

>How about any ganeti machines?
That I am not sure.

Comment 5

[Amy Rich [:arr] [:arich]]

Can you try a kernel upgrade on kvm1.build.mtv1.mozilla.com (unless there's some reason not to)?

I added bkero as a cc since he might have more knowledge about this kernel upgrade on other ganeti hosts since that was his purview until recently.

Comment 6

[Ben Kero [:bkero]]

This happens on almost every box we have, it's not a KVM specific thing.  Before rebooting, confirm that the new kernel you'll boot to has a drbd.ko module in /lib/modules.  If that exists, there should be no problem rebooting the boxes.

The problem with the drbd module is that Redhat wants to charge us an extra license to use it, and since that hasn't been something we've established with them, we're using CentOS's build-on-demand drbd kernel module package.  This has already been set up in puppet, and I've transitioned all of the existing drbd boxes to use this.

The 'audit: ' errors are harmless, although I cannot confirm if they will be fixed with a reboot.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Thanks, Ben!  Yeah, we know the errors happen everywhere and are harmless, but they're also voluminous and tend to obscure other errors, so it'd be nice to be rid of them.

Comment 8

[Brian Hourigan [:digi]]

A workaround has been implemented in bug 720239. Does rsyslog suppression satisfy your immediate requirements?

My understanding is that this is a bug in the audit implementation, and we are waiting on RedHat to patch and deploy this fix in a new kernel. It could take awhile.

Comment 9

[Amy Rich [:arr] [:arich]]

suppression wfm for the time being if you guys are rolling out the rsyslog changes via puppet (the ones in scl1 aren't rhel, so would need to be done by hand, I guess).  my main concern is that real errors get drowned in the logs now, and they take up quite a bit of space.

Comment 10

[Shyam Mani [:fox2mike]]

Don't see any further action here.