Closed Bug 1345455 Opened 7 years ago Closed 7 years ago

Send probe expiry email alerts earlier

Categories

(Data Platform and Tools :: Monitoring & Alerting, enhancement, P2)

Product:
Data Platform and Tools
Component:
Monitoring & Alerting
Type:
enhancement
Points:
1

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: gfritzsche, Unassigned)

References

Details

(Whiteboard: [measurement:client:tracking]) 

The current expiry alerts are triggered rather late, apparently 1 day before merge day (and resulting version bumps).
This leaves no time for engineers to react to changes and results in regular tree failures.

Let's:
- send those alerts earlier (proposing 6 weeks / one release cycle)
- make sure they are also sent to all "alert_emails" entries for the probe

Relatedly, we still need to make sure to send alert emails for scalars (bug 1306623) and events (bug 1345452).
Note for the triage:
This is a real issue right now, causing sheriffs & engineers to jump in & fix things when this happens around merge day.
Points: --- → 1
Priority: -- → P2
Component: Metrics: Pipeline → Monitoring & Alerting
Product: Cloud Services → Data Platform and Tools
We send these emails two weeks in advance. Long enough?
Flags: needinfo?(gfritzsche)
I'm just gonna say 2 weeks is long enough :)
Status: NEW → RESOLVED
Closed: 7 years ago
Flags: needinfo?(gfritzsche)
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.