783340 - Integrate metlog into AMO/Mkt

Status: RESOLVED FIXED

(Marketplace Graveyard :: General, defect)

Description

[Wil Clouser [:clouserw]]

https://wiki.mozilla.org/Services/Sagrada/Metlog is an awesome all-in-one data aggregator which means we won't need to juggle the (multiple!) syslog methods, CSP logger, statsd, etc. and as a bonus we don't have to deal with CEF formatting.

This bug is to replace all the current logging on https://github.com/mozilla/zamboni with metlog.  That seems like a giant project but hopefully some grep and search and replace can get us most of the way there.

Also, we should continue to have syslog routes since there will be libraries expecting them to exist.

Comment 1

[Wil Clouser [:clouserw]]

Also please read and consider bug 7650097 and bug 762942 when doing this. Thanks!

Comment 2

[Wil Clouser [:clouserw]]

Kicking to next week.  Current plan:


1) Make sure IT is setting up logstash daemon somewhere
2) Land metlog libraries in zamboni and any other dependent parts (like adding a config var to the settings file, probably putting a connection test on https://addons.mozilla.org/services/monitor, etc)
3) Create a "log to metlog" button on the "generate error" admin page so IT can make sure #1 is up and running
4) Convert the statsd logging to metlog since it is relatively centralized and simple
5) Continue converting portions of the site and landing them in production as they are completed

Comment 3

[Rob Miller [:rmiller]]

(In reply to Wil Clouser [:clouserw] from comment #2)
> Kicking to next week.  Current plan:
> 
> 
> 1) Make sure IT is setting up logstash daemon somewhere

Marked Bug 784722 as a blocker.

> 2) Land metlog libraries in zamboni and any other dependent parts (like
> adding a config var to the settings file, probably putting a connection test
> on https://addons.mozilla.org/services/monitor, etc)
> 3) Create a "log to metlog" button on the "generate error" admin page so IT
> can make sure #1 is up and running

Both of these points are addressed by the following in-progress pull request: https://github.com/mozilla/zamboni/pull/398

> 4) Convert the statsd logging to metlog since it is relatively centralized
> and simple

Currently implemented on a branch, will be queued up for pull request after initial metlog integration has landed: https://github.com/crankycoder/zamboni/commits/vng/local

> 5) Continue converting portions of the site and landing them in production
> as they are completed

TBD

Comment 4

[Wil Clouser [:clouserw]]

Thanks Rob.  Bumping to next week to continue reviews

Comment 5

[Wil Clouser [:clouserw]]

Still waiting on IT in blocking bug.  Bumping to next milestone.

Comment 6

[Wil Clouser [:clouserw]]

Hey Rob, can you give a status update on this?  Looks like the blockers are closed.

Comment 7

[Victor Ng [:vng] [:crankycoder]]

We've got one blocker left on the sentry messages.  I can see messages get to the logstash1 instance, but they're getting routed into HDFS from what I can see.

I'm pretty sure that this is a problem with the configuration of the HDFS tagger, I'm going to try reproducing the error and then file another bug to modify the logstash configuration tonight.

Comment 8

[Wil Clouser [:clouserw]]

Thanks Victor.  Let me know if I can do something (including getting others to help you if you need it).

Comment 9

[Wil Clouser [:clouserw]]

Any updates on this bug?

Comment 10

[Victor Ng [:vng] [:crankycoder]]

bugs 811861 and 792436 cover this bug.

Basically - the stock raven client comes with a Django application which effectively acts as middleware to intercept all messages.  The django-raven-metlog package adds equivalent functionality so that exceptions are intercepted and routed through raven which then delegates to metlog for transport.

Comment 11

[Victor Ng [:vng] [:crankycoder]]

The django-raven-metlog package got added to zamboni and we should be intercepting all sentry messages now.

Commit: https://github.com/mozilla/zamboni/commit/54aa3c1