Stop logging blank "events" via CEF to Arcsight

VERIFIED FIXED

Status

--
major
VERIFIED FIXED
8 years ago
7 years ago

People

(Reporter: mcoates, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [infrasec:cef][ws:moderate])

Attachments

(1 attachment)

Arcsight is receiving blank line CEF entries. We need to identify the root cause and eliminate the blank cef entries.

Here is an example. The blank "event" is between "Report" and "5" below.

Jan  5 19:55:17 wp-jpake01 /usr/bin/gunicorn: Jan 05 19:55:17 wp-jpake01.phx.weave.mozilla.com 
                 CEF:0|mozilla|keyexchange|1.3|Report| |5|cs1Label=requestClientApplication cs1=Mozilla/5.0 (X11; Linux i686; 
                 rv:2.0b8) Gecko/20100101 Firefox/4.0b8 requestMethod=POST request=/report src=10.10.14.200 
                 dest=setup.services.mozilla.com
Group: client-services-security → infra
Group: services-infra
Just to clarify the event is not entirely blank.  We are getting fields populated by the information above.  It just so happens that the field immediately following |Report| is not being populated with anything.

So it may just be that we're not populating all the fields properly or maybe there is a misalignment.
Blocks: 624025
No longer blocks: 608039
Group: infra, services-infra → mozilla-corporation-confidential

Comment 2

8 years ago
Created attachment 502137 [details] [diff] [review]
proposed patch

I believe the problem is how logs are generated for the report() function in wsgiapp.py . Under certain circumstances a log message with just a newline is logged.

http://hg.mozilla.org/services/server-key-exchange/file/2b8553b6a354/keyexchange/wsgiapp.py#l309

If the X-KeyExchange-Log header and request body are not set, then log becomes '' + '\n' .

The proposed patch removes the newline and makes the log message more verbose. There may be changes to the exact log message.
That's a client bug: it should not call the /report API without anything to report... 

Although for the server side I need to issue a 400 in that case. Will fix this.
I will double check this in Home.
* added the 400 in : http://hg.mozilla.org/services/server-key-exchange/rev/f494f05cf9e5

* will work with Richard so stage/prod are deployed with the latest tags
Home always POSTs to /report but it never POSTs any body. Instead it puts the log message int he X-KeyExchange-Log header. I thought this was the correct behaviour for short messages. If not, I can change this easily.
We should translate all \r\n in submitted reports to the literal strings "\r" or "\n", so that they aren't wrapped when sent to Arcsight.

Also note that transmission to Arcsight is done via syslog/udp and so will be truncated after a few hundred bytes, maybe as high as 1500 bytes.  It's okay to write longer messages to the log anyways, but I just wanted to mention that.
(In reply to comment #5)
> * added the 400 in :
> http://hg.mozilla.org/services/server-key-exchange/rev/f494f05cf9e5
> 
> * will work with Richard so stage/prod are deployed with the latest tags

deployed to stage as of a few minutes ago, but will be deploying again once this is fully resovled in hg
There are no "\r\n" generated on the Python side, but I can filter the message I receive to avoid them in case they occur.

What is the exact rule in ArcSight about this btw ? "\n" is a valid EOL, but if ArcSight does not support EOLs I can convert them to a specific string that's easy to process, like ### or whatever.

Another solution is to simply base64() the message if this works for Infrasec people.

Last, the message is truncated to 2000 bytes per the specis. Let me know the exact limit we should have, so I can set the same in the specs and in the code.
I am adding another bug for the '\r\n' + size issues: bug 626663

Can anyone confirm that there's no more blank in the logs ?
I setup a stage account and also generated an alert via 
https://stage-setup.services.mozilla.com/new_channel

I'm not seeing any blank lines in the log. However, I don't know how to reproduce the original blank line issue to really verify this.
Also, we are still seeing blank line events in production, but I'm assuming that's because we haven't pushed the fix yet.
(In reply to comment #11)
> I setup a stage account and also generated an alert via 
> https://stage-setup.services.mozilla.com/new_channel
> 
> I'm not seeing any blank lines in the log. However, I don't know how to
> reproduce the original blank line issue to really verify this.

The blank lines were generated by the client app that was calling /report without any report.

So you can simulate this by calling "POST /report" as Firefox Home did. You should get a 400 in this case and no log in CEF.
I confirmed that new lines are not sent.

I also noticed that the POST data is sent to CEF within the NAME field. Was this intended?

For example:
POST /report HTTP/1.1
<snip>

test


Will generate a CEF event with name=test. This could be confusing if a user flooded us with legitimate looking event names in the POST message.
Note that the /report API cannot be called if you don't know one of the channel session ID. So it cannot be flooded/attacked by random messages and will contain only codes defined in our spec.

That said, I can definitely move it to msg= and just keep in the title a fixed name like "Pake report"
I was able to craft a custom HTTP request for /report that generated CEF events without a channel session id or any auth.

Here is the POST that generated a CEF event named "456"

POST /report HTTP/1.1
Host: stage-setup.services.mozilla.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0b9) Gecko/20100101 Firefox/4.0b9
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
X-Behavioral-Ad-Opt-Out: 1
X-Do-Not-Track: 1
Pragma: no-cache
Cache-Control: no-cache
Content-Length: 3

456
(In reply to comment #15)
> That said, I can definitely move it to msg= and just keep in the title a fixed
> name like "Pake report"

Yes, that would be preferred. The message name should be static and not user influenced.
Michael, I believe this can be closed for the /report CEF issue. Can you check it on stage and RESOLVE this bug ?

Thanks
Michael, as defined in https://wiki.mozilla.org/Services/Sync/SyncKey/J-PAKE the report API is a free API that can be called without any auth or channel id.

I remember that we had a discussion about whether we should restrict it, and we said back then it was fine for now.

So, as long as you call /report with something to report, you will generate a CEF log with it. The question is whether you want to keep that behavior or not.
(In reply to comment #19)
> Michael, as defined in https://wiki.mozilla.org/Services/Sync/SyncKey/J-PAKE
> the report API is a free API that can be called without any auth or channel id.
> 
> I remember that we had a discussion about whether we should restrict it, and we
> said back then it was fine for now.
> 
> So, as long as you call /report with something to report, you will generate a
> CEF log with it. The question is whether you want to keep that behavior or not.

That's fine. Just didn't want the blank events in Arcsight. We're good here now.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
This has been verified in stage. Marking verified since the issue is confirmed in stage and is not an existing issue in prod.
Status: RESOLVED → VERIFIED
(In reply to comment #3)
> That's a client bug: it should not call the /report API without anything to
> report... 

I disagree. Also, please file a bug on the client if you think there's a client bug!

> Although for the server side I need to issue a 400 in that case. Will fix this.

Why was this introduced? The spec [1] did not mention this behaviour previously.

[1] https://wiki.mozilla.org/Services/Sync/SyncKey/J-PAKE#Server_API
(In reply to comment #22)
> Why was this introduced? The spec [1] did not mention this behaviour
> previously.

More precisely, the spec as it were agreed upon by both client and server folks: https://wiki.mozilla.org/index.php?title=Services/Sync/SyncKey/J-PAKE&oldid=280839
(In reply to comment #22)
> (In reply to comment #3)
> > That's a client bug: it should not call the /report API without anything to
> > report... 
> 
> I disagree.
 Also, please file a bug on the client if you think there's a client
> bug!

I thought we all agreed on /report behavior but it seems not. I explained the rationale in https://bugzilla.mozilla.org/show_bug.cgi?id=649535#c12 , feel free to discuss it there 


> 
> > Although for the server side I need to issue a 400 in that case. Will fix this.
> 
> Why was this introduced? The spec [1] did not mention this behaviour
> previously.
> 
> [1] https://wiki.mozilla.org/Services/Sync/SyncKey/J-PAKE#Server_API

Yes, the spec was vague about the fact that /report expects something to report to ArcSight, because it was kind of obvious. The /report status raison d'etre is to inform security via ArcSight on events that makes a transaction incomplete.  

The fact that /report closes the channel is just a extra feature to speed up the channel deletion, but /report is not an API to close channel, it's a report API.
This bug is closed ages ago. Why are we discussing things here? Please move discussion to a new or open bug.
(In reply to comment #25)
> This bug is closed ages ago. Why are we discussing things here? Please move
> discussion to a new or open bug.

Yup, sorry, my bad. I just wanted to leave a xref pointer to bug 649535 but failed to do so.
Whiteboard: [infrasec:cef][ws:moderate]

Updated

7 years ago
Group: mozilla-corporation-confidential
You need to log in before you can comment on or make changes to this bug.