965915 - send cef events to syslog's local4

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), defect)

Description

[[e-mail] (:bhearsum)]

While we were verifying bug 880358 kang noticed that we're sending cef events to the wrong log, which means they don't propagate to opsec's log machines. We should be sending these to local4.

Comment 1

[[e-mail] (:bhearsum)]

Kang, do you remember which syslog facility the events are going to right now? I just read over the code and AFAICT it should be going to local4 already...

Comment 2

[Guillaume Destuynder [:kang] [this account is no longer active]]

i'm not sure anymore, but if you ssh on the host, look at /etc/rsyslog.conf to see which facilities are forwarded to syslog servers. it was one that isn't.

otherwise, to check its sent i use tcpdump (tcpdump -i <interface> -s 1500 -XX port 514)
I think the message is currently forwarded to /var/log/messages (also set in rsyslog.conf)

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Guillaume Destuynder [:kang] (use NEEDINFO!) from comment #2)
> i'm not sure anymore, but if you ssh on the host, look at /etc/rsyslog.conf
> to see which facilities are forwarded to syslog servers. it was one that
> isn't.

I don't have to the machine, unfortunately. Chris, maybe you can help me out? There should be a bunch of CEF messages somewhere in syslog for aus4-admin.mozilla.org. I'm just looking to find out which facility they're currently being sent to.

Comment 4

[Chris Turra [:cturra]]

the following log facilities are configured to log to /var/log/message...

[[e-mail]1 ~]# grep "/var/log/messages" /etc/rsyslog.conf
*.info;local3.none;mail.none;authpriv.none;cron.none;local2.none;local5.none;local6.none;local7.!*	    /var/log/messages


if it'd be helpful, i can run the tcpdump :kang suggests above, while you generate a CEF message. just let me know.

Comment 5

[[e-mail] (:bhearsum)]

Chris and I looked at this today. We confirmed that CEF events are being generated, and that local4 is being forwarded:
13:08 <cturra> the last cef message i see was from ~5 hours ago
13:08 <cturra> Mar  3 05:14:44 ausadm mod_wsgi: Mar 03 05:14:44 ausadm.private.phx1.mozilla.com CEF:0|Mozilla|Balrog|0.5|Bad input|Bad 
               input|6|cs1Label=requestClientApplication cs1=python-requests/0.10.8 requestMethod=PUT 
               request=/releases/Thunderbird-comm-central-nightly-20140303030204/builds/WINNT_x86-msvc/be src=10.8.75.207 dhost=aus4-admin.mozilla.org suser=ffxbld 
               cs3Label=errors cs2Label=release cs2=Thunderbird-comm-central-nightly-20140303030204 cs3=Co
13:08 <bhearsum> from balrog?
13:08 <bhearsum> oh, cool
13:08 <bhearsum> so that means the app is logging stuff
13:09 <cturra> it is indeed
13:09 <cturra> [[e-mail]1 ~]# du -sh /var/log/ausadmin.log
13:09 <cturra> 694MI/var/log/ausadmin.log
13:09 <cturra> brb
13:10 <bhearsum> so...is this enough to say that it's being sent to the opsec servers?
13:13 <cturra> it looks to be sent to the syslog server from those tcpdumps
13:14 <bhearsum> ok
13:14 <bhearsum> let's call it fixed then!