While we were verifying bug 880358 kang noticed that we're sending cef events to the wrong log, which means they don't propagate to opsec's log machines. We should be sending these to local4.
Kang, do you remember which syslog facility the events are going to right now? I just read over the code and AFAICT it should be going to local4 already...
i'm not sure anymore, but if you ssh on the host, look at /etc/rsyslog.conf to see which facilities are forwarded to syslog servers. it was one that isn't. otherwise, to check its sent i use tcpdump (tcpdump -i <interface> -s 1500 -XX port 514) I think the message is currently forwarded to /var/log/messages (also set in rsyslog.conf)
(In reply to Guillaume Destuynder [:kang] (use NEEDINFO!) from comment #2) > i'm not sure anymore, but if you ssh on the host, look at /etc/rsyslog.conf > to see which facilities are forwarded to syslog servers. it was one that > isn't. I don't have to the machine, unfortunately. Chris, maybe you can help me out? There should be a bunch of CEF messages somewhere in syslog for aus4-admin.mozilla.org. I'm just looking to find out which facility they're currently being sent to.
the following log facilities are configured to log to /var/log/message... [firstname.lastname@example.org ~]# grep "/var/log/messages" /etc/rsyslog.conf *.info;local3.none;mail.none;authpriv.none;cron.none;local2.none;local5.none;local6.none;local7.!* /var/log/messages if it'd be helpful, i can run the tcpdump :kang suggests above, while you generate a CEF message. just let me know.
Chris and I looked at this today. We confirmed that CEF events are being generated, and that local4 is being forwarded: 13:08 <cturra> the last cef message i see was from ~5 hours ago 13:08 <cturra> Mar 3 05:14:44 ausadm mod_wsgi: Mar 03 05:14:44 ausadm.private.phx1.mozilla.com CEF:0|Mozilla|Balrog|0.5|Bad input|Bad input|6|cs1Label=requestClientApplication cs1=python-requests/0.10.8 requestMethod=PUT request=/releases/Thunderbird-comm-central-nightly-20140303030204/builds/WINNT_x86-msvc/be src=10.8.75.207 dhost=aus4-admin.mozilla.org suser=ffxbld cs3Label=errors cs2Label=release cs2=Thunderbird-comm-central-nightly-20140303030204 cs3=Co 13:08 <bhearsum> from balrog? 13:08 <bhearsum> oh, cool 13:08 <bhearsum> so that means the app is logging stuff 13:09 <cturra> it is indeed 13:09 <cturra> [email@example.com ~]# du -sh /var/log/ausadmin.log 13:09 <cturra> 694MI/var/log/ausadmin.log 13:09 <cturra> brb 13:10 <bhearsum> so...is this enough to say that it's being sent to the opsec servers? 13:13 <cturra> it looks to be sent to the syslog server from those tcpdumps 13:14 <bhearsum> ok 13:14 <bhearsum> let's call it fixed then!