Closed Bug 628417 Opened 13 years ago Closed 13 years ago

WARNING CSP Violation Report outputting to CEF for security

Categories

(addons.mozilla.org Graveyard :: Code Quality, defect, P2)

Product:
addons.mozilla.org Graveyard
Component:
Code Quality
Type:
defect

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: clyon, Assigned: nmiller)

References

Details

(Whiteboard: [infrasec:2011-q1 goal][qa-]) 

It would be great to get CSP violations outputted to CEF and sent to local.4 on the pm-app-amoXX systems. Once that is done, we can then pick up that data and send to ArcSight for trending.
I think this is an IT bug to adjust this in syslogd.  These are logged as z.csp.
Assignee: nobody → server-ops
Group: client-services-security → websites-security
Component: Administration → Server Operations
Product: addons.mozilla.org → mozilla.org
QA Contact: administration → mrz
Version: unspecified → other
we really want amo to output in a specific format called CEF and not syslog. we can help you guys understand this and I am pretty sure here is a cef library built for python. 

Here is a link w/ more details: https://wiki.mozilla.org/Security/Users_and_Logs
Ah, you want us to change the format.  I'll take this back to AMO then.

Meeting about this is a good idea (patches would be better).  I've had this in the back of my mind for a long time, but I find the documentation confusing.
Assignee: server-ops → nobody
Group: websites-security
Component: Server Operations → Code Quality
Product: mozilla.org → addons.mozilla.org
QA Contact: mrz → code-quality
Version: other → unspecified
I think tarek has a library for CEF logging.
We do: http://hg.mozilla.org/services/server-core/file/tip/services/cef.py

This module will let you log CEF in syslog or in a file, and uses data from the environ and a bit of configuration.

We don't have any doc yet, but I'd be happy to document it and to make it a complete standalone module -- or even release it on its own if it helps.
mcoates clarified the format/template.  The sample (https://wiki.mozilla.org/Security/CSP/Specification#Violation_Report_Sample) in CEF would be:

CEF:0|Mozilla|AMO|1.0|100|csp-report|1|src=1.2.3.4 dst=2.1.2.2 requestMethod=GET request=http://somesite.com/index.html?arg=123 requestClientApplication=Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.3a5pre) Gecko/20100601 Minefield/3.7a5pre cs1Label=blocked-uri cs1=http://evil.com/some_image.png cs2Label=violated-directive cs2=img-src 'self' cs3Label=original-policy cs3=allow 'none'; img-src *; allow 'self'; img-src 'self'"

that should be enough to add this logging.  Putting this in Q12011 for AMO because it's marked as a Q1 goal for infrasec.  If that's not the case please let me know so I can kick it out.
Target Milestone: --- → Q1 2011
Priority: -- → P2
Since this is a q1 goal for infrasec let's pull it in.  The current CSP log is a simple log() in apps/amo/views.py.  Let's adjust it to use this format.  If you're feeling fancy, you can point it at a new syslog like we do with z.timer to keep it from clogging up our log files, but that's not required.
Assignee: nobody → amckay
Target Milestone: Q1 2011 → 5.12.12
(In reply to comment #7)
> Since this is a q1 goal for infrasec let's pull it in.  The current CSP log is
> a simple log() in apps/amo/views.py.  Let's adjust it to use this format.  If
> you're feeling fancy, you can point it at a new syslog like we do with z.timer
> to keep it from clogging up our log files, but that's not required.

Nice. Let us know when it is up and running in stage or prod and we can connect up the syslog to our monitoring app.
The syslog is going to have a lot of messages in it that don't conform to this format that you should ignore.  So, feel free to hook up to it any time.
Depends on: 636102
Nate: This bug is awaiting your work to forward syslog to one of our connectors.
I've isolated the cef logger at http://hg.mozilla.org/services/cef

Andy, before I push it to PyPI, we need to work together on how you will call it.

Right now it's aimed at a wsgi app and takes a wsgi environ dict and a config dict, so that might not be suited for usage in Django or the likes.

Can you provide something similar to environ, or should we define explicit params ?  -- catch me on IRC or by Email !
The average CSP report in testing is coming in around 994 chars in testing, the max limit for syslog is 1024 (after chatting to Tarek). That's not going to leave us much room.
(In reply to comment #12)
> The average CSP report in testing is coming in around 994 chars in testing, the
> max limit for syslog is 1024 (after chatting to Tarek). That's not going to
> leave us much room.

Adding Richard S. in the loop for feedback : he asked me to set a max size because the message could get truncated at some point in the pipe before it lands into the ArcSight box. But maybe it depends in the environ so we could make this limit configurable
After chatting with mcoates, removing http headers from the report.
At 1024 bytes you should be fine. We ended up using that as our soft limit, because it's nice and safe.  Here's some of the background on that decision:

Syslog UDP packets are restricted to 1500 bytes, minus various overhead, so practically you coud push it as high as 1400 bytes before risking truncation.  This matters because the remote syslog won't reassemble two UDP packets into a single CEF line, and so potentially the CEF received by Arcsight will be damaged. So I advise truncating the "freeform text" segment to make the final UDP packet fit within the ethernet limit. This is also a reason to consider "get CEF logs to arcsight without using udp packets".
Cool thanks. I think we're good, we've got plenty of wiggle room now we've stripped the http headers.
Target Milestone: 5.12.12 → 6.0.0
Coming through to a seperate CSP log, will be double checking the format works.

https://github.com/jbalogh/zamboni/commit/16198f0a59b2e2d643bb2afd0f672e8fcc009599
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Thanks Andy, this is awesome.  Logs are on cm-vpn01 with all the other AMO logs.  Let us know if you can't find them.  (next/production will appear as we push this out, but preview is there right now)
mcoates, do you mind verifying this bug? Thanks!
(In reply to comment #19)
> mcoates, do you mind verifying this bug? Thanks!

nmmiller will need to sign off on this.

@nmiller, do we see them?
(In reply to comment #18)
> Logs are on cm-vpn01 with all the other AMO
> logs.  Let us know if you can't find them.  (next/production will appear as we
> push this out, but preview is there right now)

I looked for these under in cm-vpn01 and couldn't find them.  Any ideas where they are?  I looked under /data/logs which is linked to /mnt/netapp/nfs and there aren't any current logs in there.  Any thoughts?  Are we forwarding these CEF messages from the AMO servers anywhere else besides cm-vpn01?  Usually they are sent to local4 which is forwarded to one of our CEF syslog collectors.
Web logs are at cm-vpn01:/var/log/clusterlogs.  You'll want the /http_app_addons_csp/ directory.  The app is logging to local7, but I don't know if syslogd is doing more with them.
OK thanks...I do see the messages populate when I test against it.  What specific servers are sourcing the CEF messages?  If you could make those servers send just the CEF messages to local4 and then I can forward local4 to our collectors and we should be good.
(In reply to comment #23)
> OK thanks...I do see the messages populate when I test against it.  What
> specific servers are sourcing the CEF messages?  If you could make those
> servers send just the CEF messages to local4 and then I can forward local4 to
> our collectors and we should be good.

All of the AMO boxes are generating the logs.  I don't know if logging to local4 will work, as the logs are all combined before writing to disk and I don't know how that works (that's all IT's configuration).
Whiteboard: [infrasec:2011-q1 goal] → [infrasec:2011-q1 goal][qa-]
If it is a possibility to log to local4 like they are doing in weave/sync that would be ideal.  Then I can control what we're taking into ArcSight as we have no use for the other log data.  Here is an example of how they are doing it...Thanks!

See line 83:
http://hg.mozilla.org/services/server-key-exchange/file/f4caefc1965e/etc/keyexchange.conf#l83
Changed to LOCAL4 in https://github.com/jbalogh/zamboni/commit/fe8b01feea57494ec533c4cea20da243b93be2c1

Please verify it's doing what you want on preview: addons.allizom.org.
So we're getting the messages now, but I spoke with mcoates and it looks like we're missing some information.  Looks like part of the information is not being properly tied to a CEF field.  Here is a message:

"Mar 10 18:07:17 unused-10-2-11-236 http_app_addons_csp: z.csp:WARNING CEF:0|Mozilla|addons|1|Violation|Violation|4|cs1Label=requestClientApplication cs1=curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3 requestMethod=POST request=/services/csp/report src=63.245.220.240 dest=addons.allizom.org suser=AnonymousUser foo=bar :/data/amo_python/www/preview/zamboni/apps/amo/views.py:360"

I believe that the foo=bar part of the message at the end is supposed to be included in one of the CEF fields.  I'll let mcoates elaborate.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Target Milestone: 6.0.0 → Q1 2011
Yes. Take a look at comment 6. Every non-standard field needs to be added with
a cs label. This occurs in the format of naming the label (cs#Label=SOMENAME)
and then assigning the value (cs#=SOMEVALUE). The CEF event we received did
this properly for cs1 which contained the user agent.

This is what we need to do for foo = bar. It would be cs2Label=foo and cs2=bar.
 In the real report we would use custom labels 2 and 3 to report the core
pieces of the CSP violation (see comment 6 for example).
Andy is out next week, but this sounds like something in the CEF library.  You can see the simple call we're making to it in the commit in comment 17.
Can we just use the pypi CEF code that Tarek wrote? The more people on the standard code the better.

http://pypi.python.org/pypi/cef/0.2
...we are.  Did you look at that commit?
:) Clearly I'm not looking close enough at things.  I'll dig deeper and try to figure this out.
Tarek,

Is the CEF code library setup to accomodate more than 1 custom string?  The CEF spec allows up to 6 custom strings to be defined (cs1-cs6).  Could that be the root cause here?
Target Milestone: Q1 2011 → 6.0.3
There's no limit, its just that I guess we need additional formatting on the data I pass in. I will add that in.
https://github.com/jbalogh/zamboni/commit/a4f88365b7794b88b8e57e825b6db8d36c4ca479
https://github.com/jbalogh/zamboni-lib/commit/b75b090343da191a41bdba2ff5ae5ce02a32bd12

Let's see how that looks.
Status: REOPENED → RESOLVED
Closed: 13 years ago → 13 years ago
Resolution: --- → FIXED
Looks like the messages aren't formatting correctly still:

<164>http_app_addons_csp: z.csp:WARNING CEF:0|Mozilla|addons|1|Violation|Violation|4|src=165.182.186.147 dest=addons.mozilla.org requestClientApplication=Mozilla/5.0 (Windows NT 6.1; rv:2.0) Gecko/20100101 Firefox/4.0 requestMethod=POST request=/services/csp/report suser=AnonymousUser cs1=i cs2=e cs3=l cs1Label=v cs2Label=r cs3Label=b :/data/amo_python/www/prod/zamboni/apps/amo/views.py:360


When we use the test you gave us, curl -d "{\"csp-report\":{\"foo\":\"bar\"}}" https://addons.allizom.org/services/csp/report,
 we also see differently formatted messages, but I don't know if that's because its non prod.

<164>http_app_addons_preview_csp: z.csp:WARNING CEF:0|Mozilla|addons|1|Violation|Violation|4|src=63.245.220.240 dest=addons.allizom.org requestClientApplication=curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3 requestMethod=POST request=/services/csp/report suser=AnonymousUser :/data/amo_python/www/preview/zamboni/apps/amo/views.py:360
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
We are getting closer. The main issue right now is that the custom strings aren't mapping correctly.
I changed it so we whitelist and just 'blocked-uri', 'violated-directive', 'original-policy' are passed through to the log. A more suitable test is now:

curl -d "{\"csp-report\":{\"blocked-uri\":\"http://badsite.org\"}}" https://addons.allizom.org/services/csp/report

Locally I get:

http_app_addons_csp: z.csp:WARNING CEF:0|Mozilla|addons|1|Violation|Violation|4|src=127.0.0.1 dest=addons.mozilla.local:8000 requestClientApplication=curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3 requestMethod=POST request=/services/csp/report suser=AnonymousUser cs1Label=blocked-uri cs1=http://badsite.org :/Users/andy/sandboxes/zamboni/apps/amo/views.py:360
Assignee: amckay → nmiller
Can't access the logs on preview anymore, I think because of the syslog change. Can you try the updated curl test on it please and let me know how it goes?
The data is being correctly parsed in arcsight (yay!).

I have a couple questions on available data.
We get the following key data:

blocked-uri
https://www.paypalobjects.com/js/external/dg.js

and

Violated Directive:
script-src https://addons.allizom.org https://addons-cdn.allizom.org https://api-secure.recaptcha.net https://www.google.com :/data/amo_python/www/preview/zamboni/apps/amo/views.py:360


Is it possible to also send the URL of the page the where this occurred?  For example: The user goes to http://amo/somepage?blah&arg=<script>alert(1)</script> and this generates a violation.  I'd love to get the full page the user was on too (e.g. http://amo/somepage?blah&arg=<script>alert(1)).

Is this data available in the CSP violation report?
> data/amo_python/www/preview/zamboni/apps/amo/views.py:360

^^ that is telling you where it occured.  We could send more info in a custom field, but we run the risk of overflow again
(In reply to comment #41)
> > data/amo_python/www/preview/zamboni/apps/amo/views.py:360
> 
> ^^ that is telling you where it occured.  We could send more info in a custom
> field, but we run the risk of overflow again

That tells you where the logging occurred. We should remove that from CEF logs, it's a useless field.

The URL is included in CSP reports, I remember seeing them in our logs.
There's a request field in the CSP reports that has that, will add in. The request in the log isn't too useful since that's the location of the CSP report script.
https://github.com/jbalogh/zamboni/commit/098cbc056f0e6c1bf825f650883dc6301f7543af
Status: REOPENED → RESOLVED
Closed: 13 years ago → 13 years ago
Resolution: --- → FIXED
Status: RESOLVED → VERIFIED
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.