792436 - route sentry stacktraces through metlog

Status: RESOLVED FIXED

(Marketplace Graveyard :: General, defect)

Description

[Victor Ng [:vng] [:crankycoder]]

metlog can send stacktraces to sentry.  change all the raven calls to route through metlog instead.

Comment 1

[Wil Clouser [:clouserw]]

Victor, is this something you can do?

Comment 2

[Victor Ng [:vng] [:crankycoder]]

Hm... this should have been working in dev

https://github.com/mozilla/zamboni/blob/master/sites/dev/settings_mkt.py#L136

and

https://github.com/mozilla/zamboni/blob/84310079139d46b9f0fb5445e66a8061690d603b/sites/dev/settings_base.py#L224

I'll follow up with ops as it looks like sentry messages aren't going through the the metlog-raven plugin.

Comment 3

[Victor Ng [:vng] [:crankycoder]]

The dev instance of marketplace looks like it's not loading the correct Sentry DSN URL.

I've removed the username/password from the sentry DSN URL below, but you can see that the project ID for sentry is set to "/2/"  - that's the addons-dev project id for Sentry.

Basically - zamboni is not using the settings_local_mkt.py configuration. It looks like it's getting the addons-dev sentry dsn from /data/www/addons-dev.allizom.org/zamboni/settings_local.py instead of /data/www/addons-dev.allizom.org/zamboni/settings_local_mkt.py

Here's the METLOG_CONF in play which shows that we're using the addons-dev sentry dsn:

[[e-mail]1 zamboni]$ pwd
/data/www/addons-dev.allizom.org/zamboni
[[e-mail]1 zamboni]$ /data/www/addons-dev.allizom.org/venv/bin/python manage.py shell
Python 2.6.6 (r266:84292, Sep 12 2011, 14:03:14) 
[GCC 4.4.5 20110214 (Red Hat 4.4.5-6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.conf import settings
>>> settings.METLOG_CONF
{'logger': 'addons-dev', 'sender': {'host': ['logstash1', 'logstash2'], 'class': 'metlog.senders.UdpSender', 'port': 5566}, 'plugins': {'cef': ('metlog_cef.cef_plugin:config_plugin', {}), 'raven': ('metlog_raven.raven_plugin:config_plugin', {'dsn': 'udp://some_user:[e-mail]:9000/2'})}}
>>>

Comment 4

[Jason Thomas [:jason]]

I believe you will need to pass --settings settings_local_mkt to load marketplace configs:

-bash-4.1$ ../venv/bin/python manage.py shell_plus --settings=settings_local_mkt
Python 2.6.6 (r266:84292, Sep 12 2011, 14:03:14) 
[GCC 4.4.5 20110214 (Red Hat 4.4.5-6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> settings.METLOG_CONF
{'logger': 'addons-marketplace-dev', 'sender': {'host': ['logstash1', 'logstash2'], 'class': 'metlog.senders.UdpSender', 'port': 5566}, 'plugins': {'cef': ('metlog_cef.cef_plugin:config_plugin', {}), 'raven': ('metlog_raven.raven_plugin:config_plugin', {'dsn': 'udp://username:[e-mail]:9000/3'})}}

Comment 5

[Victor Ng [:vng] [:crankycoder]]

The logstash.conf file on logstash1 and logstash2 should change the catchall configuration to be:

    catchall {
        tags => [ "output_text", "output_statsd", "output_sentry", "output_cef" ]
        add_tag => ["output_hdfs"]
    }

After modifying /etc/logstash.conf, the logstash processes will need to be restarted.

Comment 6

[Victor Ng [:vng] [:crankycoder]]

:oremj, :jthomas - can we get this configuration change pushed to the dev instances?

Comment 7

[Jeremy Orem [:oremj]]

Pushed.

Comment 8

[Victor Ng [:vng] [:crankycoder]]

verified that metlog's messages of type 'sentry' are routing through logstash and into the proper sentry project.

Comment 9

[Victor Ng [:vng] [:crankycoder]]

bug 811861 causes stacktraces to route directly from the raven client to the sentry server.

Need to update zamboni to use http://pypi.python.org/pypi/django-raven-metlog/ to get raven to route messages through metlog

Comment 10

[Victor Ng [:vng] [:crankycoder]]

Zamboni has been updated to use django-raven-metlog as of https://github.com/mozilla/zamboni/commit/54aa3c1