https://wiki.mozilla.org/Services/Sagrada/Metlog is an awesome all-in-one data aggregator which means we won't need to juggle the (multiple!) syslog methods, CSP logger, statsd, etc. and as a bonus we don't have to deal with CEF formatting. This bug is to replace all the current logging on https://github.com/mozilla/zamboni with metlog. That seems like a giant project but hopefully some grep and search and replace can get us most of the way there. Also, we should continue to have syslog routes since there will be libraries expecting them to exist.
Also please read and consider bug 7650097 and bug 762942 when doing this. Thanks!
Kicking to next week. Current plan: 1) Make sure IT is setting up logstash daemon somewhere 2) Land metlog libraries in zamboni and any other dependent parts (like adding a config var to the settings file, probably putting a connection test on https://addons.mozilla.org/services/monitor, etc) 3) Create a "log to metlog" button on the "generate error" admin page so IT can make sure #1 is up and running 4) Convert the statsd logging to metlog since it is relatively centralized and simple 5) Continue converting portions of the site and landing them in production as they are completed
(In reply to Wil Clouser [:clouserw] from comment #2) > Kicking to next week. Current plan: > > > 1) Make sure IT is setting up logstash daemon somewhere Marked Bug 784722 as a blocker. > 2) Land metlog libraries in zamboni and any other dependent parts (like > adding a config var to the settings file, probably putting a connection test > on https://addons.mozilla.org/services/monitor, etc) > 3) Create a "log to metlog" button on the "generate error" admin page so IT > can make sure #1 is up and running Both of these points are addressed by the following in-progress pull request: https://github.com/mozilla/zamboni/pull/398 > 4) Convert the statsd logging to metlog since it is relatively centralized > and simple Currently implemented on a branch, will be queued up for pull request after initial metlog integration has landed: https://github.com/crankycoder/zamboni/commits/vng/local > 5) Continue converting portions of the site and landing them in production > as they are completed TBD
Thanks Rob. Bumping to next week to continue reviews
Still waiting on IT in blocking bug. Bumping to next milestone.
Hey Rob, can you give a status update on this? Looks like the blockers are closed.
We've got one blocker left on the sentry messages. I can see messages get to the logstash1 instance, but they're getting routed into HDFS from what I can see. I'm pretty sure that this is a problem with the configuration of the HDFS tagger, I'm going to try reproducing the error and then file another bug to modify the logstash configuration tonight.
Thanks Victor. Let me know if I can do something (including getting others to help you if you need it).
Any updates on this bug?
bugs 811861 and 792436 cover this bug. Basically - the stock raven client comes with a Django application which effectively acts as middleware to intercept all messages. The django-raven-metlog package adds equivalent functionality so that exceptions are intercepted and routed through raven which then delegates to metlog for transport.
The django-raven-metlog package got added to zamboni and we should be intercepting all sentry messages now. Commit: https://github.com/mozilla/zamboni/commit/54aa3c1