Closed Bug 626663 Opened 14 years ago Closed 14 years ago

CEF log max size and content

Categories

(Cloud Services :: Server: Core, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: tarek, Assigned: tarek)

Details

Attachments

(1 file)

From Richard: We should translate all \r\n in submitted reports to the literal strings "\r" or "\n", so that they aren't wrapped when sent to Arcsight. Also note that transmission to Arcsight is done via syslog/udp and so will be truncated after a few hundred bytes, maybe as high as 1500 bytes. It's okay to write longer messages to the log anyways, but I just wanted to mention that.
There are no "\r\n" generated on the Python side, but I can filter the message I receive to avoid them in case they occur. What is the exact rule in ArcSight about this btw ? "\n" is a valid EOL, but if ArcSight does not support EOLs I can convert them to a specific string that's easy to process, like ### or whatever. Another solution is to simply base64() the message if this works for Infrasec people. Last, the message is truncated to 2000 bytes per the specis. Let me know the exact limit we should have, so I can set the same in the specs and in the code.
I don't think "\r" or "\n" are going to survive the log->local4->arcsight process intact. I would suggest convert newlines to the two-character string '\n' so that it's clear what they were. I would like to request a spec limit of 1024 bytes total including both the log header and the log post body.
Multi-line fields can be sent by CEF by encoding the newline character as \n or \r. Note that multiple lines are only allowed in the value part of the extensions. For example: Sep 19 08:26:10 host CEF:0|security|threatmanager|1.0|100|Detected a threat. No action needed.|10|src=10.0.0.1 msg=Detected a threat.\n No action needed.
You can find full details on the CEF specifics here: https://intranet.mozilla.org/File:CEF_White_Paper.pdf
Will change the function accordingly: - allow newlines only on extensions - define a default maxsize of 1024 for the whole log.
Assignee: nobody → tarek
Attachment #510270 - Flags: review?(telliott)
Comment on attachment 510270 [details] [diff] [review] Patch that implements the CEF maxsize + escaping Looks good, though it looks really weird to have an unescaped pipe in a regex :P
Attachment #510270 - Flags: review?(telliott) → review+
(In reply to comment #7) > Comment on attachment 510270 [details] [diff] [review] > Patch that implements the CEF maxsize + escaping > > Looks good, though it looks really weird to have an unescaped pipe in a regex > :P hehe well, in a set of characters ([]), special characters are not active. You just just need to escape [, ] and -. Sometimes it's not even necessary (if it's the first character of the set. See: http://docs.python.org/library/re.html#regular-expression-syntax
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: