1487523 - Define the SLO requirements for the production logging CEP system

Status: RESOLVED INVALID

(Data Platform and Tools :: Monitoring & Alerting, enhancement)

Description

[Mike Trinkala [:trink]]

What is currently deployed clearly does not match the expectations of what the service level objectives should be causing some frustration. We need to define those expectations so a more ideal setup can be put in place. For general operational monitoring the current setup has been 'good enough' but the security monitoring requirements are clearly different. Generally operational alerts are rare but with security monitoring (logins etc) we are alerting all the time and a few minute delay (like during a deploy) with a quick back-fill is no longer acceptable. Let's define the SLO's for items below and how the service should report and gracefully degrade if they aren't being met.

# Up time
The current setup in not deployed to be redundant with hot fail over. This means every time we push a new release (minimum of every two weeks) it causes a several minute outage (hours per year); add in actual issues and unplanned outages and at best we only have three nines of up time.

# State Preservation on Shutdown/Restart
State preservation is another issue, in general the state is written to an ephemeral disk and it is treated as such (temporary scratch pad). Technically this should have the same reliability as the instance it is running on but has always been best effort (if it is a different version or we are running out of space or the re-load fails, we drop it, if we are starting a new instance we generally let it start from scratch (no state migration)).

# Aggregate Loss
Related to state loss as the aggregates are also written to ephemeral storage. These are the local graphs, JSON output, etc. If the data is needed it has to be configured to be written back to pub/sub or persistent store otherwise it will be lost when the instance is terminated.

# Message Loss
The infrastructure is designed to have an 'at least once' delivery guarantee but there are still ways to skip (lose data).
## Instance Termination (no hot fail over)
  - any data that was pulled down by the instance but not fully processed before termination is lost
  - Kinesis - all new instances are configured to start from the current time. Any data between the termination and the start of the new instance will not be processed (this is not a issue on GCP and Kafka pub/sub as they track the positioning via a consumer group/subscription and not a local checkpoint file on the ephemeral disk)
## Configuration Errors
  - Mis-configurations can cause plugins to ignore data (e.g an incorrect message matcher)

# Message Duplication
The infrastructure is designed to to checkpoint after processing. So in a failure case it can replay the data since the last checkpoint duplicating up to one second of data (tens of thousands of messages with our current data rate).

# Message Latency
In general message latency should only be seconds from the time it was created but it is highly source dependent i.e. reading from CloudTrail data has a 10-15 minute latency.  Another issue affecting latency is the capacity of the CEP instance. As we add more data and workload we will eventually have to upgrade the instance hardware and/or partition the data.

Comment 1

[Aaron Meihm [:alm]]

Marking invalid as part of larger project refocus to migrate from hindsight/CEP to GCP Dataflow/Stackdriver