1412014 - release-notifications: Stop sending email when a release task completed

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation: Other, enhancement)

Description

[Johan Lorenzo [:jlorenzo]]

In bug 1388680, we decided to kill IRC notifications sent by release-notifications (aka pulse-notify).

A few days ago, :sfraser, :bhearsum, :catlee and I thought this would a good idea not send email about succeeded tasks anymore. The main rationale is: These emails create noise and provide low value. On the other hand, links to failed tasks are actionable. If needed, it's easy to find what completed upstream task made one task fail, thanks to the task inspector.

I personally have a filter that marks completed tasks as read. After a year, I realize I've never needed a look into any completed task -- from my Inbox. However, deleting ~12 months of notifications (170.000+ threads) turned out to be painful on Gmail.

How do the other releaseduty people feel about it? Does anybody have a use case to share?

Comment 1

[Kim Moir [:kmoir] ET]

+1 to less email

Comment 2

[Rail Aliiev [:rail]]

(noise--)++ :)

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

+1

Comment 5

[Aki Sasaki (not active)]

Less email is good. I think it might be good to know if things are hung, but if we're not finding that via email, then it's just noise.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I have a similar filter setup as you Johan, but don't look at success emails. The only use case for them I can think of is our switch to the newer scheduler API, when the task group explorer didn't handle old task group IDs any more. We have http://mozilla-release-logs.s3-website-us-east-1.amazonaws.com/ to work around that, so no objections from me for disabling success emails.

If we do this, I predict we will also want to suppress emails when a graph is cancelled.

Comment 7

[Jordan Lund (:jlund)]

+1 :)

Comment 8

[Rok Garbas [:garbas]]

+1 for the change. Would it make sense to wait until we have everything in-tree? and then creating a patch for this?

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

Found in triaging bug 1374497 which is related. Might be worth discussing this as well in order to reduce even more our inbox churn. tl;dr: "when canceling graphs we end up with 1000+ "Exception" emails in our inboxes. Might be worth tweaking logic for non-expired and non-canceled scenarios only".

Comment 10

[Johan Lorenzo [:jlorenzo]]

Attachment: releasetasks PR

It can make sense to wait until relpro is in tree. The patch in releasetasks is pretty straightforward, though. If it doesn't interfere with the tc-migration, feel free to pick it.

Comment 11

[Rail Aliiev [:rail]]

Comment on attachment 8924925 [details] [review]
releasetasks PR

... with a nit

Comment 12

[Johan Lorenzo [:jlorenzo]]

Landed on master https://github.com/mozilla-releng/releasetasks/commit/de376c25f9538b7628fdc267a807747076b513b0

This will get live at next deployment.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

We didn't anticipate that we'd lose logs as part of this, eg http://mozilla-release-logs.s3-website-us-east-1.amazonaws.com/mozilla-beta/firefox-58.0b5/build1/. That's a serious regression for future archaeology. Is there some way we can block emails but not logs ?

Comment 14

[Johan Lorenzo [:jlorenzo]]

Great catch, Nick! Sorry for the lack of logs. I reverted attachment 8924925 [details] [review] in https://github.com/mozilla-releng/releasetasks/commit/570c30e25c114b9465074aedf32d63620ad3dbd1. I deployed that commit on both bm83 and bm85. 

We may be able to deploy the missing logs by fetching them from TC and uploading them manually. Nick, would you know how to get a set of creds that has write access to mozilla-release-logs.s3-website-us-east-1.amazonaws.com? 

Apart from that, there's very likely a way to stop email notifications from pulse-notify. I'll look into it more closely.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

I ended up changing my mind on this in bug 1425571 when implementing notifications for task failures/exceptions via taskcluster-notify. No messages for successful completion there, and we'll use task artifacts for logs as they have sufficient longevity for practical purposes. If you have cycles to reland for 58 and 52ESR then go ahead. Apologies for the churn.

Comment 16

[Johan Lorenzo [:jlorenzo]]

I think we can leave it as is on 58 and 52esr. release-notifications will be retired after we kill esr52. Closing bug as WONTFIX.