960274 - Add filters and notifiers to heka for statsd data

Status: RESOLVED INCOMPLETE

(Cloud Services :: Operations: Marketplace, task, P4)

Description

[Jason Thomas [:jason]]

Create Lua filters that will watch the statsd data as it flows through, and generate notifiers when the triggers happen.

Comment 1

[Jeremy Orem [:oremj]]

What data do we want to filter/notify on?

Comment 2

[Rob Miller [:rmiller]]

Andy: Have specs for the first few stats you'd like to notify on?

Comment 3

[Andy McKay]

I like the standard deviation script that you showed in the secret channel for these values:

stats.addons.response.200
stats.addons.response.301
stats.addons.response.400
stats.addons.response.401
stats.addons.response.403
stats.addons.response.500

Comment 4

[Rob Miller [:rmiller]]

Great, I'll start with these. Will try to have something to experiment with by the end of this week.

Comment 5

[Rob Miller [:rmiller]]

Aaaaannnd I'm not going to have anything ready by the end of this week, I've been busy w/ other things. I've finally been able to start on it today, though, and will definitely have some stuff to try out next week.

Comment 6

[Rob Miller [:rmiller]]

I've made a ton of progress on this and am to a point where I'm ready to start testing out my code on the running dev/stage Heka instance. To simplify the testing and iteration process, I've had [e-mail] update the Heka config to allow me to dynamically add and remove sandbox filters w/o a restart or any further config changes. My first attempt at uploading my script failed, though, b/c I was depending on recent changes to the Lua environment, so Heka itself needs to be updated first. I've built an RPM from a recent Heka nightly and have given it to Jason for deployment, attached here for posterity.

Comment 7

[Rob Miller [:rmiller]]

Okay, well, the RPM was slightly too large to upload, so here's a link:

https://people.mozilla.org/~rmiller/heka/heka-0_5_0-20140205-linux-amd64.rpm

Comment 8

[Rob Miller [:rmiller]]

We have lift-off:

* http://logstash1.addons.phx1.mozilla.com:4352/#plugins/filters/dev_sbxmgr-http_monitor

Apologies for the still buggy dashboard UI, but this page might need a refresh before you see everything. Ultimately, however, there should be a list of output links on the right side of the page. Among these should be 4 of type "CBUF", which will take you to graphs for the response codes for each listed host.

For now there are no notifiers going out. Instead, when an anomaly is detected, an annotation is placed on the graph. This is to avoid a flood of false positives, since we have a lot of work to get the anomaly detection dialed in to not fire except when we really want it to. This is especially true when we're on dev and stage, since the traffic is low and intermittent.

I need to switch gears for a bit to handle some other Heka stuff, but I'll come back to this in a week or so to help push this along further. The code that's generating those graphs was dynamically loaded into Heka, so for now I'm going to attach that code and the related config to this bug so we have it for posterity.

Comment 9

[Rob Miller [:rmiller]]

Attachment: Lua source to the addons/marketplace http_monitor filter

Comment 10

[Rob Miller [:rmiller]]

Attachment: TOML config for addons/marketplace http_monitor filter

Comment 11

[Jeremy Orem [:oremj]]

Rob, is https://github.com/mozilla-services/heka/blob/dev/sandbox/lua/filters/http_status.lua equivalent?

Comment 12

[Rob Miller [:rmiller]]

It's close. The filter you linked to set up to process the output of Heka's http server log (nginx, apache, haproxy) parsing instead of the same info coming in via statsd.

I actually did deploy a filter to graph and analyse the statsd data. The anomaly detection stuff is only doing graph annotations, not actually sending notifiers, b/c at the time our anomaly detection code was a bit more rough and had too many false positives when the data was sparse. We've since added some algorithms that will handle this better.

Also, I've recently added a filter that can handle statsd data in the general case:

https://github.com/mozilla-services/heka/blob/dev/sandbox/lua/filters/stat_graph.lua

which is a better choice than the one-off that I originally wrote for marketplace. Just realized I need to add support for anomaly detection there, though, I'll work on that today or tomorrow and will update this ticket when it's ready.

Comment 13

[Rob Miller [:rmiller]]

Okay, as of https://github.com/mozilla-services/heka/pull/980 the stat_graph filter (for general purpose graphing and monitoring of specific statsd data points) supports anomaly detection and has been merged to dev. So the custom filter code that I deployed can be replaced, either by http_status.lua (if you want to go the 'parsing the log files' route) or by stat_graph.lua (if you want to continue to use statsd as the source of the monitored data).

Comment 14

[Jeremy Orem [:oremj]]

Added dev/stage HTTPStatus report here: http://kibana1.stage.addons.phx1.mozilla.com:4352/#sandboxes/HTTPStatus/outputs/HTTPStatus.HTTPStatus.cbuf