Send probe expiry email alerts earlier

RESOLVED WORKSFORME

Status

P2
normal
RESOLVED WORKSFORME
2 years ago
11 months ago

People

(Reporter: gfritzsche, Unassigned)

Tracking

Details

(Whiteboard: [measurement:client:tracking])

(Reporter)

Description

2 years ago
The current expiry alerts are triggered rather late, apparently 1 day before merge day (and resulting version bumps).
This leaves no time for engineers to react to changes and results in regular tree failures.

Let's:
- send those alerts earlier (proposing 6 weeks / one release cycle)
- make sure they are also sent to all "alert_emails" entries for the probe

Relatedly, we still need to make sure to send alert emails for scalars (bug 1306623) and events (bug 1345452).
(Reporter)

Comment 1

2 years ago
Note for the triage:
This is a real issue right now, causing sheriffs & engineers to jump in & fix things when this happens around merge day.

Updated

2 years ago
Points: --- → 1
Priority: -- → P2

Updated

2 years ago
Component: Metrics: Pipeline → Monitoring & Alerting
Product: Cloud Services → Data Platform and Tools

Comment 2

a year ago
We send these emails two weeks in advance. Long enough?
Flags: needinfo?(gfritzsche)

Comment 3

11 months ago
I'm just gonna say 2 weeks is long enough :)
Status: NEW → RESOLVED
Last Resolved: 11 months ago
Flags: needinfo?(gfritzsche)
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.