853920 - [socorro-crashstats] Implement statsd on the internal middleware

Status: RESOLVED FIXED

(Socorro :: Webapp, task)

Description

[Peter Bengtsson [:peterbe]]

We should record:

* number of requests made to the middleware
** # 2xx codes
** # 3xx codes
** # 4xx codes
** # 5xx codes
* time it takes to make the requests

(:lonnen, should we count each error (like 301) code or reduce it to groups like 3xx)

Most likely the tech to use is: https://github.com/andymckay/django-statsd

Comment 1

[Lonnen :lonnen]

metrics need to be prefixed appropriately

socorro-{env}.{component}.rest.of.the.metric

^- from env   ^- hardcoded

Comment 2

[Lonnen :lonnen]

peter -- I believe the new status is a dupe of 908334? The new analytics middleware should capture /api/ as well.

This was becoming a bug to track the same stats from the django-middleware. If that's not wanted we can close this.

Comment 3

[Peter Bengtsson [:peterbe]]

Sorry. I'm a mess. 

I didn't know about #908334.

I saw that this bug was about Socorro::Webapp but the bug summary talked about middleware. 

What we need, is statsd in both places. Or all three??

1) The middleware (done by adding statsd to middleware_app.py)
2) The django 
(optionally 3) Where django makes requests to the middleware

I guess I was thinking we'd put in statsd calls on the public API which reflect what the it gets back from the middleware. 

Now I think that, since we lightly proxy the middleware in the public API we can simply have a general statsd thing for ALL django requests. 

What makes it confusing is that we use the middleware in two places in the webapp. We use it to generate the templates and we use it for the public API. 

We can either slap a generic statsd on ALL django requests but sometimes what we do is get one code from the middleware (e.g. 404) but swallow that and respond with (200 - Sorry Crash ID not found). 

Does that make any sense?

I guess we need a new bug. Something like: "Implemented statsd for middleware" and then we'll let #908334 deal with all statsd for all things Django. Then we can close this bug as invalid.

Comment 4

[Lonnen :lonnen]

So 908334 is landed. It keeps count of requests by verb, response code, and path. It should catch public api calls.


For this bug, lets get similar tracking in the middleware. It could be as simple as copying over my django middleware from the front end.

Comment 5

[Peter Bengtsson [:peterbe]]

(In reply to Chris Lonnen :lonnen from comment #4)
> 
> For this bug, lets get similar tracking in the middleware. It could be as
> simple as copying over my django middleware from the front end.

You mean, in crashstats/crashstats/models.py ?

Comment 6

[Gabriela Thumé]

Ok, so we don't want statsd at 'views' anymore? (public API)

I am a little bit confused about the internal middleware. Where should I deal with statsd for it?

Comment 7

[Lonnen :lonnen]

There's a django version of the socorro middleware in /socorro/ somewhere (not to be confused with webapp-django). That django instance should have a django-middleware applied to it.

Socorro uses the term middleware to mean intermediary api. In django a middleware is code that is called on in the middle of the request-response process.

Comment 8

[Gabriela Thumé]

I am sorry to bother you about it again but I don't know what django version are you talking about. I didn't found any script django related in /socorro/ :(

I saw that the status response is raised at places like https://github.com/mozilla/socorro/blob/master/socorro/middleware/middleware_app.py#L436, should I create the statsd in there? (Is the only place I know where I can get the status)

Comment 9

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/e65dc02656e034ef654d8d556873fa6f86958d37
Fixes bug 853920 - statsd on internal middleware