1464142 - Add monitoring for Sentry rate limiting for Nightly JS Errors project

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Osmose [:osmose, :mkelly]]

The Nightly JS Errors project in Sentry is hitting some extreme ratelimiting, and in bug 1464138 we're trying to reduce the amount of requests we're sending from clients. In order to know whether we're affecting the rate of incoming requests, we need a way to see:

- The count of incoming requests over time
- The count of requests that were rejected due to ratelimiting over time

Wezhou, would you be able to set up monitoring for these two events in a place I could view them while working on this issue, like Datadog?

Comment 1

[:wezhou]

The datadog-nginx integration only provides limited number of metrics according to the document: https://docs.datadoghq.com/integrations/nginx/

Specifically, it can only tell us about "the count of incoming requests over time". For the other one, it looks like we need to parse the access log ourselves and then send it to datadog.

I'll do some research and see if can come up with a solution.

Comment 2

[:wezhou]

We have added datadog integration to the sentry cluster.

It is now possible to see http.request.count metric in datadog with the following tags:

App: sentry
Type: web
Env: prod
Status: 2XX or 4XX

Let us know if you have questions.

Comment 3

[Osmose [:osmose, :mkelly]]

The measure has been in Datadog for a while, and measuring 400s vs 200s covers the need in the summary, so this is resolved. Thanks!