If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Integrate metlog into AMO/Mkt

RESOLVED FIXED

Status

Marketplace
General
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: clouserw, Assigned: RaFromBRC)

Tracking

Points:
---
Dependency tree / graph

Details

(Reporter)

Description

5 years ago
https://wiki.mozilla.org/Services/Sagrada/Metlog is an awesome all-in-one data aggregator which means we won't need to juggle the (multiple!) syslog methods, CSP logger, statsd, etc. and as a bonus we don't have to deal with CEF formatting.

This bug is to replace all the current logging on https://github.com/mozilla/zamboni with metlog.  That seems like a giant project but hopefully some grep and search and replace can get us most of the way there.

Also, we should continue to have syslog routes since there will be libraries expecting them to exist.
(Reporter)

Updated

5 years ago
Blocks: 783382
(Reporter)

Comment 1

5 years ago
Also please read and consider bug 7650097 and bug 762942 when doing this. Thanks!
Assignee: nobody → rmiller
(Reporter)

Updated

5 years ago
Target Milestone: --- → 2012-08-23
(Reporter)

Updated

5 years ago
Target Milestone: 2012-08-23 → 2012-08-30
(Reporter)

Comment 2

5 years ago
Kicking to next week.  Current plan:


1) Make sure IT is setting up logstash daemon somewhere
2) Land metlog libraries in zamboni and any other dependent parts (like adding a config var to the settings file, probably putting a connection test on https://addons.mozilla.org/services/monitor, etc)
3) Create a "log to metlog" button on the "generate error" admin page so IT can make sure #1 is up and running
4) Convert the statsd logging to metlog since it is relatively centralized and simple
5) Continue converting portions of the site and landing them in production as they are completed

Updated

5 years ago
Blocks: 785924
(Assignee)

Updated

5 years ago
Depends on: 784722
(Assignee)

Comment 3

5 years ago
(In reply to Wil Clouser [:clouserw] from comment #2)
> Kicking to next week.  Current plan:
> 
> 
> 1) Make sure IT is setting up logstash daemon somewhere

Marked Bug 784722 as a blocker.

> 2) Land metlog libraries in zamboni and any other dependent parts (like
> adding a config var to the settings file, probably putting a connection test
> on https://addons.mozilla.org/services/monitor, etc)
> 3) Create a "log to metlog" button on the "generate error" admin page so IT
> can make sure #1 is up and running

Both of these points are addressed by the following in-progress pull request: https://github.com/mozilla/zamboni/pull/398

> 4) Convert the statsd logging to metlog since it is relatively centralized
> and simple

Currently implemented on a branch, will be queued up for pull request after initial metlog integration has landed: https://github.com/crankycoder/zamboni/commits/vng/local

> 5) Continue converting portions of the site and landing them in production
> as they are completed

TBD
(Reporter)

Comment 4

5 years ago
Thanks Rob.  Bumping to next week to continue reviews
Target Milestone: 2012-08-30 → 2012-09-06
(Reporter)

Comment 5

5 years ago
Still waiting on IT in blocking bug.  Bumping to next milestone.
Target Milestone: 2012-09-06 → 2012-09-13
Depends on: 791072
(Reporter)

Comment 6

5 years ago
Hey Rob, can you give a status update on this?  Looks like the blockers are closed.
Target Milestone: 2012-09-13 → ---
We've got one blocker left on the sentry messages.  I can see messages get to the logstash1 instance, but they're getting routed into HDFS from what I can see.

I'm pretty sure that this is a problem with the configuration of the HDFS tagger, I'm going to try reproducing the error and then file another bug to modify the logstash configuration tonight.
Depends on: 792436
(Reporter)

Comment 8

5 years ago
Thanks Victor.  Let me know if I can do something (including getting others to help you if you need it).
(Reporter)

Updated

5 years ago
Blocks: 799666
(Reporter)

Comment 9

5 years ago
Any updates on this bug?
bugs 811861 and 792436 cover this bug.

Basically - the stock raven client comes with a Django application which effectively acts as middleware to intercept all messages.  The django-raven-metlog package adds equivalent functionality so that exceptions are intercepted and routed through raven which then delegates to metlog for transport.
The django-raven-metlog package got added to zamboni and we should be intercepting all sentry messages now.

Commit: https://github.com/mozilla/zamboni/commit/54aa3c1
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Depends on: 822759
You need to log in before you can comment on or make changes to this bug.