1635856 - Monitor the error rate of the ingestion edge

Status: NEW

(Data Platform and Tools :: Monitoring & Alerting, task, P2)

Description

[Mark Reid [:mreid]]

This could be implemented in the delivery checks dash similar to the existing latency monitor.

Comment 1

[Ben Wu [:benwu]]

Is this metric being recorded somewhere or does it need to be instrumented from scratch?

Comment 2

[Daniel Thorn [:relud]]

the necessary metrics are visible here on the stackdriver load balancer resource page[0], but that doesn't tell what metrics it's actually using. I was able to get response by code class in the metrics explorer[1], but not response ratio by code class.

[0] https://console.cloud.google.com/monitoring/dashboards/resourceDetail/l7_lb_rule,project_id:moz-fx-data-ingesti-prod-579d,url_map_name:k8s-um-prod-data-ingestion-data-ingestion-prod-data-ingestion-0?project=moz-fx-data-ingesti-prod-579d&timeDomain=1h

[1] https://console.cloud.google.com/monitoring/metrics-explorer?project=moz-fx-data-ingesti-prod-579d&pageState=%7B%22xyChart%22:%7B%22dataSets%22:%5B%7B%22timeSeriesFilter%22:%7B%22filter%22:%22metric.type%3D%5C%22loadbalancing.googleapis.com%2Fhttps%2Frequest_count%5C%22%20resource.type%3D%5C%22https_lb_rule%5C%22%22,%22minAlignmentPeriod%22:%2260s%22,%22unitOverride%22:%221%22,%22aggregations%22:%5B%7B%22perSeriesAligner%22:%22ALIGN_RATE%22,%22crossSeriesReducer%22:%22REDUCE_SUM%22,%22groupByFields%22:%5B%22metric.label.%5C%22response_code_class%5C%22%22%5D%7D,%7B%22crossSeriesReducer%22:%22REDUCE_NONE%22%7D%5D%7D,%22targetAxis%22:%22Y1%22,%22plotType%22:%22LINE%22%7D%5D,%22options%22:%7B%22mode%22:%22COLOR%22%7D,%22constantLines%22:%5B%5D,%22timeshiftDuration%22:%220s%22,%22y1Axis%22:%7B%22label%22:%22y1Axis%22,%22scale%22:%22LINEAR%22%7D%7D,%22isAutoRefresh%22:true,%22timeSelection%22:%7B%22timeRange%22:%221h%22%7D%7D