1380353 - Monitor/Report/Alert 'heavy hitters' by ID

Status: RESOLVED FIXED

(Cloud Services :: Fraud Detection, enhancement)

Description

[Mike Trinkala [:trink]]

Rough requirements:
1) ID is configurable/selectable from the data (e.g. IP address, token, client ID) 
2) some false positives are acceptable (will start with a rate of 0.00012 and tune as necessary)
3) configurable sliding window, minimum resolution of one minute, support for at least 100MM entries
4) have an accessible list of heavy hitters (top 1000) for review even if no alerts are being generated
5) facets are determined by message selection e.g (by service, protocol, status code) 
6) configurable alert thresholds based on historical behavior. As this would have to be per ID more clarification is needed on how this is managed/maintained.

Comment 1

[Mike Trinkala [:trink]]

This is a work in progress (need to get some measurements on the heavy hitter table management). Also, this implementation uses the expiring cuckoo filter so the false positive rate is much lower ~2 in a billion.

Simon #6 needs to be defined (this example should provide a good starting point for a discussion)

https://github.com/mozilla-services/lua_sandbox_extensions/pull/153/commits/41adb103d8f2f25c0bbc7868ec9889b1744ef9f3

Comment 2

[Simon Bennetts [:psiinon]]

This is all looking good to me :)
As per our irc chat, I think #6 can be defered: we need to see what sort of results we get out of this and then decide on a per service basis what we need to do. In some cases alerts might be appropriate, but we only want to get people out of bed if theres really something useful (and important) they can do.

Comment 3

[Mike Trinkala [:trink]]

https://github.com/mozilla-services/lua_sandbox_extensions/blob/master/moz_security/sandboxes/heka/analysis/moz_security_heavy_hitters.lua

including a tigerblood output

https://github.com/mozilla-services/lua_sandbox_extensions/blob/master/moz_security/sandboxes/heka/output/moz_security_tigerblood.lua