637507 - Add formatter handler to CEF

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Code Quality, defect)

Description

[Andy McKay]

Adds support for cef as a formatter for Python logging.

Comment 1

[Andy McKay]

Attachment: Adds logging formatter support

Comment 2

[Tarek Ziadé (:tarek)]

Comment on attachment 516344 [details] [diff] [review]
Adds logging formatter support

>diff -r 27ef8b7c65ea README.txt>diff -r 27ef8b7c65ea cef.py
>--- a/cef.py	Thu Feb 24 21:49:57 2011 +0100
>+++ b/cef.py	Mon Feb 28 15:50:29 2011 -0800
>@@ -74,13 +74,13 @@
>     _SYSLOG_OPTIONS = _SYSLOG_PRIORITY = _SYSLOG_FACILITY = None
>     SYSLOG = False
> 
>+import logging
> import socket
> from time import strftime
> import re
> try:
>     from services import logger
> except ImportError:
>-    import logging
>     logger = logging.getLogger('CEF')  # NOQA
> 
> _HOST = socket.gethostname()
>@@ -203,21 +203,8 @@
>     return params
> 
> 
>-def log_cef(name, severity, environ, config, username='none',
>-            signature=None, **kw):
>-    """Creates a CEF record, and emit it in syslog or another file.
>-
>-    Args:
>-        - name: name to log
>-        - severity: integer from 0 to 10
>-        - environ: the WSGI environ object
>-        - config: configuration dict
>-        - signature: CEF signature code - defaults to name value
>-        - username: user name - defaults to 'none'
>-        - extra keywords: extra keys used in the CEF extension
>-    """
>-    # XXX might want to remove the request dependency here
>-    # so this module is standalone

Since it's only used locally let's make it a private name: _get_fields


>+def get_fields(name, severity, environ, config, username=None,
>+               signature=None, **kw):
>     name = _convert_prefix(name)
>     if signature is None:
>         signature = name
>@@ -225,7 +212,6 @@
>         signature = _convert_prefix(signature)
> 
>     severity = _convert_prefix(severity)
>-    config = _filter_params('cef', config)
>     source = _get_source_ip(environ)
> 
>     fields = {'severity': severity,
>@@ -255,12 +241,13 @@
> 
>     # overriding with provided datas
>     fields.update(kw)
>+    return fields
> 
>-    # resulting message
>-    msg = _CEF_FORMAT % fields
> 

Same here: _format_msg

>+def format_msg(fields, kw, maxlen=_MAXLEN):
>     # adding custom extensions
>     # sorting by size
>+    msg = _CEF_FORMAT % fields
>     extensions = [(len(str(value)), len(key), key, value)
>                     for key, value in kw.items()
>                   if key not in _EXTENSIONS]
>@@ -273,7 +260,7 @@
>         value = _convert_ext(value)
>         key = _check_key(key)
> 
>-        if msg_len + added_len > _MAXLEN:
>+        if maxlen and msg_len + added_len > maxlen:
>             # msg is too big.
>             warn = 'CEF Message too big. %s %s' % (msg, str(kw.items()))
>             logger.warning(warn)
>@@ -282,6 +269,27 @@
>         msg += ' %s=%s' % (key, value)
>         msg_len += added_len
> 
>+    return msg
>+
>+
>+def log_cef(name, severity, environ, config, username='none',
>+            signature=None, **kw):
>+    """Creates a CEF record, and emit it in syslog or another file.
>+
>+    Args:
>+        - name: name to log
>+        - severity: integer from 0 to 10
>+        - environ: the WSGI environ object
>+        - config: configuration dict
>+        - signature: CEF signature code - defaults to name value
>+        - username: user name - defaults to 'none'
>+        - extra keywords: extra keys used in the CEF extension
>+    """
>+    config = _filter_params('cef', config)
>+    fields = get_fields(name, severity, environ, config, username=username,
>+                        signature=signature, **kw)
>+    msg = format_msg(fields, kw)
>+
>     if config['file'] == 'syslog':
>         if not SYSLOG:
>             raise ValueError('syslog not supported on this platform')
>@@ -290,3 +298,36 @@
>         with _log_lock:
>             with open(config['file'], 'a') as f:
>                 f.write('%s\n' % msg)
>+
>+

s/level_map/LEVEL_MAP/

>+level_map = {
>+    logging.DEBUG: syslog.LOG_DEBUG,
>+    logging.WARNING: syslog.LOG_WARNING,
>+    logging.INFO: syslog.LOG_INFO,
>+    logging.ERROR: syslog.LOG_ERR,
>+    logging.CRITICAL: syslog.LOG_CRIT,
>+}
>+

s/Formatter/_Formatter/

>+
>+class Formatter(logging.Formatter):
>+    def format(self, record):
>+        kw = record.args
>+        fields = get_fields(record.msg, kw['severity'], kw['environ'],
>+                            {'version': kw.get('version', 0),
>+                             'vendor': kw.get('vendor', 'Mozilla'),
>+                             'device_version':kw.get('device_version', '1'),
>+                             'product': kw.get('product', 'Mozilla')},
>+                            username=kw.get('username'),
>+                            signature=kw.get('signature'))
>+
>+        datefmt = getattr(self, 'datefmt', None)
>+        if not datefmt:
>+            datefmt = '%H:%M:%s'
>+        fields['date'] = strftime(datefmt)
>+        return format_msg(fields, kw['data'], maxlen=kw.get('maxlen'))
>+
>+
>+class SysLogFormatter(Formatter):
>+    def format(self, record):
>+        record.args['severity'] = level_map[record.levelno]
>+        return Formatter.format(self, record)
>diff -r 27ef8b7c65ea test_cef.py
>--- a/test_cef.py	Thu Feb 24 21:49:57 2011 +0100
>+++ b/test_cef.py	Mon Feb 28 15:50:29 2011 -0800


Also, a few pep8 fixes:

$ flake8 *.py
cef.py:318:46: E231 missing whitespace after ':'
test_cef.py:176:1: E302 expected 2 blank lines, found 1
test_cef.py:206:12: E225 missing whitespace around operator


Otherwise, looks good


Otherwise

Comment 3

[Andy McKay]

Attachment: With feedback

Formatter is designed to be used externally, for where you want CEF formatting for non sys logs (which isn't used at this time) so would rather not rename. Rest sounds good.

Comment 4

[Tarek Ziadé (:tarek)]

Andy, not sure why you're assigning me the bug since you are writing the patch -- I am just the reviewer here ;)

Comment 5

[Tarek Ziadé (:tarek)]

Comment on attachment 516619 [details] [diff] [review]
With feedback

is this class is going to be used outside the module ?

If not s/Formatter/_Formatter/

>+class Formatter(logging.Formatter):
..

otherwise looks good

Comment 6

[Andy McKay]

Sorry not sure of the flow in bugzilla. I rewrote the patch and added it to the bug so you could merge and push out a new release to pypi :)

Formatter is not currently used outside the module, but is designed to be in the future.

Comment 7

[Tarek Ziadé (:tarek)]

done in http://hg.mozilla.org/services/cef/rev/70e12d250311

Thanks a lot !

Comment 8

[Tarek Ziadé (:tarek)]

released at http://pypi.python.org/pypi/cef/0.2