Common monitors/alerts for all sources ingested by pipeline (minimum set)

RESOLVED FIXED

Status

Cloud Services
Metrics: Pipeline
P1
normal
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: Katie Parlante, Assigned: trink)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
quoting :mreid... Critical for trusting data that comes out of the pipeline.

We should have a ​minimum set of ​monitors for every source coming in to the pipeline that watches the basics:

- Submission rates
- gzip errors (for data that comes in compressed)
- json parse errors (for json data)
- decoder errors (for any custom code that runs on incoming records)

Monitoring specific particular data sources should be logged as separate bugs.
(Reporter)

Updated

3 years ago
Assignee: nobody → kparlante
(Reporter)

Updated

3 years ago
Status: NEW → ASSIGNED
(Reporter)

Updated

3 years ago
Depends on: 1135252
(Reporter)

Updated

3 years ago
Group: mozilla-employee-confidential

Updated

3 years ago
Priority: -- → P1

Updated

3 years ago
Assignee: kparlante → nobody

Updated

3 years ago
Group: mozilla-employee-confidential
Assignee: nobody → kparlante

Updated

3 years ago
Depends on: 1163818
(Reporter)

Comment 3

3 years ago
We've defined the minimum set of common monitors as:
- TelemetryStats drops to zero (cep)
- TelemetryOutput ProcessFileFailures (dwl) (any increase)
- puppet configured plugin terminations
- dwl generic monitor for all plugins

Trink is implementing this so reassigning to him.
Assignee: kparlante → mtrinkala
Summary: Common monitors for all sources ingested by pipeline (minimum set) → Common monitors/alerts for all sources ingested by pipeline (minimum set)
(Assignee)

Comment 5

3 years ago
The PRs have been reviewed and merged.
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.