1748268 - `__heartbeat__` should send errors to sentry if it errors out

Status: RESOLVED FIXED

(Socorro :: Webapp, defect, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Over the weekend, we had a bunch of HTTP 500s from the /__heartbeat__ endpoint, but there isn't anything in Sentry.

When the /__heartbeat__ endpoint errors out, it really should send something to Sentry indicating the problem.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

The Socorro webapp doesn't have a /__broken__ endpoint, so I'm adding one. I need to add the infra bits first.

https://github.com/mozilla-services/cloudops-deployment/pull/4495

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 5956: bug 1748268: implement /__broken__ view and clean up heartbeat view

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #5956: "bug 1748268: implement /broken view and clean up heartbeat view" in e027151.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 5957: bug 1748268: log sentry error sanitization errors

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #5957: "bug 1748268: log sentry error sanitization errors" in f633e49.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

I added the /__broken__ endpoint and once that deployed to stage, I used it to trigger an HTTP 500. That part worked fine, however, nothing got sent to Sentry.

The webapp uses sentry-sdk and there's a bunch of sentry error sanitzation code that gets run before the errors get sent to Sentry. If the sanitization code errors out, it emits an webapp.sentry.before_send_exception incr. I looked that up and it looks like it's been erroring out for a while. However, since all it does is re-raise the exception that's getting thrown, I think that exception is getting swallowed and we never see it in the logs. I tweaked the sanitization exception handling code to log that exception and I'll see where that gets me.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

That worked out nicely:

Uncaught exception:
  File "/app/webapp-django/crashstats/sentrylib.py", line 89, in __call__
    sanitizer(event, hint)
  File "/app/webapp-django/crashstats/sentrylib.py", line 137, in __call__
    crumb = crumb_sanitizer(crumb, {})
  File "/app/webapp-django/crashstats/sentrylib.py", line 171, in __call__
    if crumb.get("category") != "query" or not crumb.get("message"):
<class 'AttributeError'>
AttributeError("'str' object has no attribute 'get'")

I'll fix that and then theoretically sanitizing will work and then we're good to go again.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 5960: bug 1748268: fix sanitizer bug

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #5960: "bug 1748268: fix sanitizer bug" in 11a79dd.

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

The last fix deployed to stage and I verified that errors are now getting reported to Sentry from the webapp in stage.

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

This was deployed an hour or so ago in bug #1748856. It should be fixed now.