Closed Bug 1450680 Opened 6 years ago Closed 6 years ago

Use Log::Log4perl::Appender::Raven to send (some) errors to Sentry

Categories

(bugzilla.mozilla.org :: Infrastructure, enhancement)

Production
enhancement
Not set
normal

Tracking

()

RESOLVED FIXED

People

(Reporter: dylan, Assigned: bobm)

References

Details

Attachments

(1 file, 2 obsolete files)

For ERROR and FATAL-level events, we should send those to sentry.

Really this bug is just about allowing the log4perl config file to be configured in that way.
Just a note that:

- if we use the 'user' or 'http' interface options (Log::Log4perl::MDC->put(sentry_user => {...}), we need to make sure to censor passwords and other private information.
- Examples of this https://gist.github.com/dylanwh/02d97f51c4a11d9fc4920e8493ce0d5f (being removed by bug 
1450679)
This hasn't been done because we'd also need to consider censoring the stack information, and as a result the cost of implementing this has been more than I'd wanted to pay.

Ideally we'd like the http and user information as well.
We won't get the http or user stuff yet, but that can be another bug.

Let's configure the stacks to log to sentry.
Component: General → Infrastructure
QA Contact: mcote
Attached file log4perl-prod.conf.erb (obsolete) —
Okay, here's a (probably correct) erb. I tested it with the 'erb' command and was able to sent entry events to my own sentry.
Note you need to use the deprecated DSN which includes the key, which will have a colon before the '@' part.

The file can be named anything, just pass LOG4PERL_CONFIG_FILE= to the container.

Let me know when this is ready on bugzilla-stage, and I'll login and manually trigger some logging errors.
Assignee: dylan → bobm
Attachment #9010395 - Attachment mime type: application/octet-stream → text/plain
(In reply to Dylan Hardison [:dylan] (he/him) from comment #2)
> This hasn't been done because we'd also need to consider censoring the stack
> information, and as a result the cost of implementing this has been more
> than I'd wanted to pay.
> 
> Ideally we'd like the http and user information as well.

Just to be clear, I had confirmation from multiple security people that this isn't a concern for us.
Comment on attachment 9010395 [details]
log4perl-prod.conf.erb

Puppet's very specific about what ERB formatting it accepts.  If I modify it the template to look like the following:

<% if @sentry_dsn -%>
log4perl.rootLogger = INFO, Socket, Screen, Sentry
<% else -%>
log4perl.rootLogger = INFO, Socket, Screen
<% end -%>

<%#
    the following three lines are required to output in the mozlog json format.
%>
<% MozLogLayout = lambda do |appender, layout| -%>
log4perl.appender.<%= appender %>.layout = Log::Log4perl::Layout::Mozilla
log4perl.appender.<%= appender %>.layout.max_json_length = 16384
log4perl.appender.<%= appender %>.layout.name = <%= layout %>
<% end -%>

log4perl.appender.Socket = Log::Log4perl::Appender::Socket
log4perl.appender.Socket.PeerAddr = 127.0.0.1
log4perl.appender.Socket.PeerPort = 5880
log4perl.appender.Socket.defer_connection=1
<%= MozLogLayout.call('Socket', 'CEREAL') %>

log4perl.filter.LOG_TO_STDERR = sub { not $ENV{LOG4PERL_STDERR_DISABLE} }
log4perl.appender.Screen = Log::Log4perl::Appender::Screen
log4perl.appender.Screen.Filter = LOG_TO_STDERR
log4perl.appender.Screen.stderr = 1
<%= MozLogLayout.call('Screen', 'STDERR') %>

<% if @sentry_dsn -%>
log4perl.appender.Sentry = Log::Log4perl::Appender::Raven
log4perl.appender.Sentry.Threshold = ERROR
log4perl.appender.Sentry.sentry_dsn = "<%= @sentry_dsn %>"
log4perl.appender.Sentry.layout = Log::Log4perl::Layout::PatternLayout
log4perl.appender.Sentry.layout.ConversionPattern=[%c] %m %n
<% end -%>

And feed it a dummy sentry_dsn value of: TestSentryDsn, I get the following output:

log4perl.rootLogger = INFO, Socket, Screen, Sentry



log4perl.appender.Socket = Log::Log4perl::Appender::Socket
log4perl.appender.Socket.PeerAddr = 127.0.0.1
log4perl.appender.Socket.PeerPort = 5880
log4perl.appender.Socket.defer_connection=1
log4perl.appender.Socket.layout = Log::Log4perl::Layout::Mozilla
log4perl.appender.Socket.layout.max_json_length = 16384
log4perl.appender.Socket.layout.name = CEREAL
log4perl.rootLogger = INFO, Socket, Screen, Sentry



log4perl.appender.Socket = Log::Log4perl::Appender::Socket
log4perl.appender.Socket.PeerAddr = 127.0.0.1
log4perl.appender.Socket.PeerPort = 5880
log4perl.appender.Socket.defer_connection=1
log4perl.appender.Socket.layout = Log::Log4perl::Layout::Mozilla
log4perl.appender.Socket.layout.max_json_length = 16384
log4perl.appender.Socket.layout.name = CEREAL


log4perl.filter.LOG_TO_STDERR = sub { not $ENV{LOG4PERL_STDERR_DISABLE} }
log4perl.appender.Screen = Log::Log4perl::Appender::Screen
log4perl.appender.Screen.Filter = LOG_TO_STDERR
log4perl.appender.Screen.stderr = 1
log4perl.appender.Screen.layout = Log::Log4perl::Layout::Mozilla
log4perl.appender.Screen.layout.max_json_length = 16384
log4perl.appender.Screen.layout.name = STDERR
log4perl.rootLogger = INFO, Socket, Screen, Sentry



log4perl.appender.Socket = Log::Log4perl::Appender::Socket
log4perl.appender.Socket.PeerAddr = 127.0.0.1
log4perl.appender.Socket.PeerPort = 5880
log4perl.appender.Socket.defer_connection=1
log4perl.appender.Socket.layout = Log::Log4perl::Layout::Mozilla
log4perl.appender.Socket.layout.max_json_length = 16384
log4perl.appender.Socket.layout.name = CEREAL
log4perl.rootLogger = INFO, Socket, Screen, Sentry



log4perl.appender.Socket = Log::Log4perl::Appender::Socket
log4perl.appender.Socket.PeerAddr = 127.0.0.1
log4perl.appender.Socket.PeerPort = 5880
log4perl.appender.Socket.defer_connection=1
log4perl.appender.Socket.layout = Log::Log4perl::Layout::Mozilla
log4perl.appender.Socket.layout.max_json_length = 16384
log4perl.appender.Socket.layout.name = CEREAL


log4perl.filter.LOG_TO_STDERR = sub { not $ENV{LOG4PERL_STDERR_DISABLE} }
log4perl.appender.Screen = Log::Log4perl::Appender::Screen
log4perl.appender.Screen.Filter = LOG_TO_STDERR
log4perl.appender.Screen.stderr = 1
log4perl.appender.Screen.layout = Log::Log4perl::Layout::Mozilla
log4perl.appender.Screen.layout.max_json_length = 16384
log4perl.appender.Screen.layout.name = STDERR


log4perl.appender.Sentry = Log::Log4perl::Appender::Raven
log4perl.appender.Sentry.Threshold = ERROR
log4perl.appender.Sentry.sentry_dsn = "TestSentryDsn"
log4perl.appender.Sentry.layout = Log::Log4perl::Layout::PatternLayout
log4perl.appender.Sentry.layout.ConversionPattern=[%c] %m %n

Does that look right?
Flags: needinfo?(dylan)
Attachment #9010395 - Flags: review?(dylan)
Attached file config file without templating (obsolete) —
Here's result of me running erb(1) which seems to speak a different dialect than puppet (grrr).

the sentry dsn is SENTRY_DSN, you can re-templatize on that?
Attachment #9010395 - Attachment is obsolete: true
Attachment #9010395 - Flags: review?(dylan)
Flags: needinfo?(dylan)
Attachment #9012428 - Flags: review?(bobm)
Attached file correct erb version
turns out the using lambdas in templates is *undefined* behavior. So here's an .erb without any lambdas.
Attachment #9012428 - Attachment is obsolete: true
Attachment #9012428 - Flags: review?(bobm)
Added.  Resolving this bug.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: