1281765 - Send reminder emails for expired probes to alert_emails recipients too

Status: RESOLVED DUPLICATE of bug 1610616

(Data Platform and Tools :: Monitoring & Alerting, defect, P3)

Description

[Georg Fritzsche [:gfritzsche]]

Currently there are apparently many expired probes and accompanying code which don't get removed.
We could send regular reminder emails about them so they are not forgotten about.

Bug 1261400 should also help here once the bug triaging process is fully in place.

Comment 1

[Chris H-C :chutten]

We send regular reminder emails but only to telemetry-alerts, not to the alerts_emails fields specifically. [1]

Not sure why (before my time) but we should figure out why we made the decision to not send it to alerts_emails and, if that reason is no longer valid, send them to alert_emails.

Comment 2

[Chris H-C :chutten]

[1]: https://github.com/mozilla/cerberus/blob/master/alert/expiring.py#L127

Comment 3

[Thomas Huelbert]

to discuss with DDurst.

Comment 4

[Thomas Huelbert]

to you, Mr Durst.

Comment 5

[David Durst [:ddurst]]

I assume the reason is because either alerts_emails isn't required? Anyway, it makes sense to me that if we have addresses there, we should send to them as well. I also assume it's not the case that everything has to be removed -- so this is just about ensuring that nothing falls through the cracks. RyanVM may have a stronger opinion on this than I do.

Comment 6

[Thomas Huelbert]

over to RyanVM

Comment 7

[Ryan VanderMeulen [:RyanVM]]

This sounds like a good idea to me, albeit with limited knowledge of the history that led to the current state of affairs.

My understanding is that this is one of the issues that data review was intended to fix going forward (having a clear idea of ownership of a probe throughout its lifecycle). It's unfortunate that we have some probes without alert_emails set, but maybe we can handle those differently as-needed (hg blame should be able to tell us who needs bugging in any case). Anyway, this sounds good to me. We don't want to be leaving a bunch of cruft from old probes laying around IMO.

Comment 8

[Georg Fritzsche [:gfritzsche]]

alerts_emails is required for all new probes (we enforce it at build time and via data collection review).

Only old probes from before we added that field have exceptions for that rule. Identifying emails/owners for all of these would be a time consuming task, which is why we still have them without.

Comment 9

[Thomas Huelbert]

to Chris, setting p2, feel free to change as needed.

Comment 10

[:shell escalante]

chutten - do you know if part of the "getting to glean -FoG plan" includes removing dead probes from the code?

If yes we can just close this out - and add as part of the "migration plan" docs for teams as they migrate.

Comment 11

[Chris H-C :chutten]

Oh goodness, this is something the Probe Info Service already does, and has done for a couple of years. Guess this bug got left behind.

Closest to a duplicate bug I can find is bug 1610616, so let's use it.