1431770 - Investigate monitorability of the GCP telemetry sample

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Comment 1

[Daniel Thorn [:relud]]

tl;dr The sample is monitorable.

leaving dataflow monitoring out of this, because it isn't included in the telemetry sample yet. investigating monitoring as part of the bug that investigates using dataflow.

stackdriver integrations built into pubsub, bigquery, and app engine contain thorough metrics, and the ability to alert on each of those metrics, within a specific subset of the service, or across the service as a whole.

Metrics available include:

pubsub subscriber undelivered messages, aged of oldest undelivered message, message volume, and latency.
pubsub topic publish message volume, cost, internal and external errors, and operation volume.
app enginge http code volumes, cpu/mem/disk usage, latency percentiles, instance count, memcache use, datastore use, and cached/static/dynamic response volumes.
bigquery query volume, query time percentiles, and slot quota availability.

The logging interface can be used to generate metrics from logs, and can also forward filtered sets of logs to a pubsub topic, cloud storage bucket, or bigquery table if desired.

I have not yet seen a way create complex monitors within stackdriver, like ones that do math on multiple metrics, or combine multiple metrics into a single alert.

Stackdriver does not appear to have a built-in export functionality, but does provide an api for reading timeseries data out. Example using this api to export data: https://github.com/frodenas/stackdriver_exporter

So we /can/ do the basics, and for more complex stuff there's no blockers per se, but they may require a decent time investment.