Monitor/Report/Alert 'heavy hitters' by ID

RESOLVED FIXED

Status

Cloud Services
Manatee
RESOLVED FIXED
9 months ago
3 months ago

People

(Reporter: trink, Assigned: trink)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Assignee)

Description

9 months ago
Rough requirements:
1) ID is configurable/selectable from the data (e.g. IP address, token, client ID) 
2) some false positives are acceptable (will start with a rate of 0.00012 and tune as necessary)
3) configurable sliding window, minimum resolution of one minute, support for at least 100MM entries
4) have an accessible list of heavy hitters (top 1000) for review even if no alerts are being generated
5) facets are determined by message selection e.g (by service, protocol, status code) 
6) configurable alert thresholds based on historical behavior. As this would have to be per ID more clarification is needed on how this is managed/maintained.
(Assignee)

Updated

9 months ago
Assignee: sbennetts → mtrinkala
Points: --- → 3
(Assignee)

Comment 1

9 months ago
This is a work in progress (need to get some measurements on the heavy hitter table management). Also, this implementation uses the expiring cuckoo filter so the false positive rate is much lower ~2 in a billion.

Simon #6 needs to be defined (this example should provide a good starting point for a discussion)

https://github.com/mozilla-services/lua_sandbox_extensions/pull/153/commits/41adb103d8f2f25c0bbc7868ec9889b1744ef9f3
Flags: needinfo?(sbennetts)
This is all looking good to me :)
As per our irc chat, I think #6 can be defered: we need to see what sort of results we get out of this and then decide on a per service basis what we need to do. In some cases alerts might be appropriate, but we only want to get people out of bed if theres really something useful (and important) they can do.
Flags: needinfo?(sbennetts)

Updated

7 months ago
Blocks: 1403572
You need to log in before you can comment on or make changes to this bug.