1467029 - heroku-release-notifications app/worker "has no notifications section" alerts

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement)

Description

[Jordan Lund (:jlund)]

recently started alerting in papertrail.

Likely due to recent puppet/releaserunner pyenv changes we made around releasetasks and requirements.


nthomas> yeah, I wonder if we accidentally never deployed https://github.com/mozilla-releng/releasetasks/commit/953dddcba9e8fe917664b6567b488d21fbfb78b2
21:57:23 then pulse-notify doesn't fail gracefully when no notification config to look up

Comment 1

[Johan Lorenzo [:jlorenzo] - PTO - Back August 24th]

I think we should retire heroku-release-notifications. These days, it's only used to send emails to the mailing list, which we don't want anymore. It's also just used by esr52.

We're still getting email alerts. As a quick fix, I turned off the heroku worker[1]. Let's agree to kill heroku-release-notifications before actually doing it. What do you guys think?

[1] I set the number of dynos on https://dashboard.heroku.com/apps/release-notifications.

Comment 2

[Johan Lorenzo [:jlorenzo] - PTO - Back August 24th]

(In reply to Johan Lorenzo [:jlorenzo] from comment #1)
> We're still getting email alerts. As a quick fix, I turned off the heroku
> worker[1].

I did the same on the dev instance: https://dashboard.heroku.com/apps/release-notifications-dev

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

We may need them still for the release signoff notifications about updates available/pushed/etc etc ?

Comment 4

[Jordan Lund (:jlund)]

If the fix is unknown and will take time, maybe we should just send explicit emails about state of esr52 releases in terms of availability/pushed/published.

I updated esr 52.9.0 prereqs to do this.

Is that good enough? Should we close?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

I agree it's not worth the time to fix pulse-notify but there's another way, which should let us re-enable the automated notifications to release-signoff without the spam for everything else.

I've tweaked the papertrail search [1] from:

ERROR -("ERROR - Task exception was never retrieved" OR "RuntimeError('Not connected',)" OR 'RuntimeError("Not connected")' OR "RuntimeError: Not connected")

to:

ERROR -("ERROR - Task exception was never retrieved" OR "RuntimeError('Not connected',)" OR 'RuntimeError("Not connected")' OR "RuntimeError: Not connected" OR "KeyError: 'notifications'" OR "NoNotificationConfigurationError" OR "has no notifications section")

which suppresses all matches from 52.8.1esr. 

We can re-enable prod heroku (not bothered about dev) and avoid manual work in sending email, and carrying forward prereqs.


[1] https://papertrailapp.com/searches/17080452/edit

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I've re-enabled prod Heroku, which cleared out the queue backlog, and removed the prereq from 52.9.0 in releasewarrior-data.

Comment 7

[Jordan Lund (:jlund)]

thanks nick!