Closed Bug 1344247 Opened 7 years ago Closed 7 years ago

Add hindsight monitor for validation error rates

Categories

(Cloud Services Graveyard :: Metrics: Pipeline, enhancement, P1)

Product:
Cloud Services Graveyard
Component:
Metrics: Pipeline
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: Dexter, Assigned: trink)

References

(Blocks 1 open bug)

Details

We want:
- email alerts to telemetry-client-dev@mozilla.com
- graphing over time, per-channel, per-type
- regression detection, per-channel, per-type
- failure samples ready for triaging

This should alert us if there's any sudden spike in the schema validation errors for any ping, e.g. the main ping fails to suddenly validate for a portion of the clients.

Having a sample of the ping which fails to validate would be very helpful to help diagnosing issues.

Bonus points for making this easy to use for testing new ping schemas.
Blocks: 1344235
Depends on: 1344249
Assignee: nobody → mtrinkala
Points: --- → 1
Priority: -- → P1
Proposed alert configuration (this would track all types of error not just schema failures):

      ingestion_errors = {
        max_error_ratio = 0.005,
      },
Blocks: 1348863
Summary: Add hindsight monitor for schema validation error rates → Add hindsight monitor for validation error rates
Product: Cloud Services → Cloud Services Graveyard
You need to log in before you can comment on or make changes to this bug.