1316431 - antenna: set up sentry for -dev

Status: RESOLVED FIXED

(Cloud Services :: Operations: Antenna, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

We want to use Sentry for capturing and alerting on unhandled exceptions in Antenna.

This bug covers setting up Antenna -dev environment with Sentry.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

I suspect we need to do code/requirements changes first, so I'll take this bug to look into that.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think we can do it with this:

https://docs.sentry.io/clients/python/integrations/wsgi/

I think we can do that and if it exists, then wrap the WSGI app. I need to look into whether Falcon lets errors propagate upward.

The other way to do this is do the captureException() call in the exception handler in Antenna here:

https://github.com/mozilla/antenna/blob/cdf810e40f0d5d4b3a93d65db43108b5df108a21/antenna/app.py#L155

If we're going to go that route, we should probably upgrade to Falcon 1.0 first which will require us to redo our error handling.

I'll look into both of those today.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Implemented the needed code in Antenna in PR: https://github.com/mozilla/antenna/pull/128

That adds a SENTRY_DSN configuration variable. Once that code lands, we can add that configuration variable to the -dev configuration and then that should work.

Comment 4

[Miles Crabill [:miles]]

Sentry is at the very least working in Stage, as can be seen here: https://sentry.prod.mozaws.net/operations/antenna-stage/.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Right on!

Last week, we discussed adding an endpoint or something so that we could verify it was working more easily. I still want to do that. Will try to do it this week.

Relatedly, I turned most/all of those into bugs to look into this week.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: Link to Github pull-request: https://github.com/mozilla/antenna/pull/130

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

I implemented the endpoint, it got deployed and I tested it on -dev just now:

https://antenna.dev.mozaws.net/__broken__

We should have gotten a Sentry email, but I haven't seen one. Further, Sentry doesn't seem to be aware of the error:

https://sentry.prod.mozaws.net/operations/antenna-dev/

Given that, I'm pretty sure Sentry is not working on -dev.

Passing this to Miles to look into it.

Comment 8

[Miles Crabill [:miles]]

Sentry is working in dev. There was an issue with secrets configuration that affected both logging and sentry, and was limited to dev. That issue should be resolved now!

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

I see the exception that popped up in Sentry. This looks good to me. Thank you!