1441400 - webapp isn't sending errors to sentry

Status: RESOLVED FIXED

(Socorro :: Webapp, task, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

I was looking into bug #1441318. I can reproduce that bug in a local dev environment. It returns an HTTP 500 page--that's not wildly helpful, but it's fine. It's probably easy to fix. However, it got me wondering about other things like why we're not seeing anything in Sentry.

Socorro has a view that simulates an unhandled exception. I triggered that in -prod and saw an HTTP 500, saw the error in the logs on the node I was on (just like what Miles put in comment #2), but nothing in Sentry.

I checked for muted and ignored errors--nothing. I checked for errors from other Socorro components like the processor and crontabber--they seem fine (which is to say, they're raising lots of errors! yay?).

The last error in Sentry that I see for the webapp is January 10th.

This bug covers looking into whether the webapp is sending errors to sentry and if not, fixing it.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm making this a P1 because if the webapp isn't sending errors, that's bad.

Comment 2

[Osmose [:osmose, :mkelly]]

Taking this to see if I can get some quick investigation done today.

Comment 3

[Osmose [:osmose, :mkelly]]

Luckily, Raven comes with a management command for testing these kinds of things. Here's my output locally running it:

```
┌─( ~/P/s/webapp-django λ:master 0f28375 )────────────────────────────────────────────────────( [socorro] @abaddon )─┐
└─❱❱❱ docker-compose run webapp bash                                                                  +10261 12:47 ❰─┘
Starting socorro_statsd_1 ...
Starting socorro_rabbitmq_1 ... done
Starting socorro_memcached_1 ... done
Starting socorro_localstack-s3_1 ... done
Starting socorro_postgresql_1 ...
Starting socorro_elasticsearch_1 ... done
app@webapp:/app$ cd webapp-django/
app@webapp:/app/webapp-django$ ./manage.py raven test
StatsdMetrics configured: statsd:8125 None
Client configuration:
  base_url       : https://sentry.prod.mozaws.net
  project        : 63
  public_key     : ********************************
  secret_key     : ********************************

Sending a test message... Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/raven/base.py", line 274, in handle_exception
    self.captureException(exc_info=exc_info, level='fatal')
  File "/usr/local/lib/python2.7/site-packages/raven/base.py", line 814, in captureException
    'raven.events.Exception', exc_info=exc_info, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/raven/contrib/django/client.py", line 297, in capture
    result = super(DjangoClient, self).capture(event_type, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/raven/base.py", line 634, in capture
    **kwargs)
  File "/usr/local/lib/python2.7/site-packages/raven/contrib/django/client.py", line 238, in build_msg
    data = super(DjangoClient, self).build_msg(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/raven/base.py", line 470, in build_msg
    for processor in self.get_processors():
  File "/usr/local/lib/python2.7/site-packages/raven/base.py", line 293, in get_processors
    yield self.module_cache[processor](self)
  File "/usr/local/lib/python2.7/site-packages/raven/base.py", line 78, in __missing__
    module, {}, {}, [class_name]), class_name)
  File "/app/webapp-django/crashstats/tokens/utils.py", line 4, in <module>
    class RavenSanitizeAuthTokenProcessor(SanitizePasswordsProcessor):
  File "/app/webapp-django/crashstats/tokens/utils.py", line 9, in RavenSanitizeAuthTokenProcessor
    tuple(SanitizePasswordsProcessor.FIELDS) + ('auth-token',)
TypeError: 'property' object is not iterable

Original exception was:
Traceback (most recent call last):
  File "./manage.py", line 12, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/raven/contrib/django/management/commands/raven.py", line 69, in handle
    'data': options.get('data'),
  File "/usr/local/lib/python2.7/site-packages/raven/scripts/runner.py", line 81, in send_test_message
    'loadavg': get_loadavg(),
  File "/usr/local/lib/python2.7/site-packages/raven/base.py", line 792, in captureMessage
    return self.capture('raven.events.Message', message=message, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/raven/contrib/django/client.py", line 297, in capture
    result = super(DjangoClient, self).capture(event_type, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/raven/base.py", line 634, in capture
    **kwargs)
  File "/usr/local/lib/python2.7/site-packages/raven/contrib/django/client.py", line 238, in build_msg
    data = super(DjangoClient, self).build_msg(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/raven/base.py", line 470, in build_msg
    for processor in self.get_processors():
  File "/usr/local/lib/python2.7/site-packages/raven/base.py", line 293, in get_processors
    yield self.module_cache[processor](self)
  File "/usr/local/lib/python2.7/site-packages/raven/base.py", line 78, in __missing__
    module, {}, {}, [class_name]), class_name)
  File "/app/webapp-django/crashstats/tokens/utils.py", line 4, in <module>
    class RavenSanitizeAuthTokenProcessor(SanitizePasswordsProcessor):
  File "/app/webapp-django/crashstats/tokens/utils.py", line 9, in RavenSanitizeAuthTokenProcessor
    tuple(SanitizePasswordsProcessor.FIELDS) + ('auth-token',)
TypeError: 'property' object is not iterable
```

Looks like it might be an upgrade issue? I get the same output when running the command on staging.

Comment 4

[Osmose [:osmose, :mkelly]]

Attachment: pr 4357: Fix bug 1441400: Replace custom exception sanitization processor.

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/1cf1ebafe7e2881bda223811c25aefde192f2e54
Fix bug 1441400: Replace custom exception sanitization processor.

Raven added a configurable processor for sanitizing error data in 6.4.0, which
removes the need for our own custom subclass. In addition, the internal API for
SanitizePasswordsProcessor changed such that our error reporting was broken.

https://github.com/mozilla-services/socorro/commit/f20a7cbea3de6c6798fb11f497a120754471eac9
Merge pull request #4357 from Osmose/raven-processor

Fix bug 1441400: Replace custom exception sanitization processor.