Closed Bug 1430772 Opened 7 years ago Closed 7 years ago

release-automation-notifications doesn't get email on failures in 59

Categories

(Release Engineering :: Release Automation, enhancement)

Product:
Release Engineering
Component:
Release Automation
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1425571

People

(Reporter: jcristau, Assigned: mtabara)

Details

In 58 the release-automation-notifications list got email for every completed and failed release task. (For a while it was only failed tasks, but that got backed out according to bug 1412014.) For 59.0b1 no emails seem to be sent, even though we had plenty of builds, and plenty of failed tasks.
Oddly enough, one of them at least arrived, some time later on: "FAILED: [devedition 59.0b1 build11/mozilla-beta] release-version-bump-devedition task"
59.0b2 was full green graph so no emails arrived - https://tools.taskcluster.net/groups/eqTuHQPvRT-Gz1-dlZWiRA. Pushing/Shipping graph were as green as `promote` one. I'll debug now the 59.0b1 where the original issues occurred.
Assignee: nobody → mtabara
First, a little bit of context. General context: * TC jobs are indexed under some certain routes * there's certain filters on those specific routes to trigger pulse messages * pulse-notify (aka release-notifications) handles processes thoses notifications and sends them up How? a) reads pulse messages via the pulseguardian account and processes them b) sends the notification over to IRC c) sends AWS SNS messages in the cloud for emails So in order for this to function properly we need: i) correct task defined with routes ii) filtering setup to trigger pulse messages upon those routes iii) release-notifications service up and running in Heroku to read those pulse messages via pulseguardian iv) process and send the notifications in irc/email via AWS SES
I started debugging from latter to former and didn't notice anything suspicious in the pulse-notify serivce in Heroku. Everything seems fine there, as well as the pulseguardian. I then checked the task definitions to make sure the email addresses are correct by-project and the notifications logic is there. All good. Then I noticed the lack of routes in some particular tasks. We've had 11 different builds for Devedition-59.0b1 of which the majority of the failing tasks were: * l10n jobs * update verification * version bump Looking close at these tasks, I noticed that these tasks were precisely the ones missing the routes: E.g. nightly-l10n failing, no routes defined [1] E.g. update verification failing, no routes defined [2] E.g. version bump failing, emails sent because routes present [3] [1]: https://tools.taskcluster.net/groups/SNGDx4l-SC6xhdUtoxG-Og/tasks/KYZOQhAdTyytX4PsBipP6w/details [2]: https://tools.taskcluster.net/groups/Neyjzg81QtKTx9M84MG3RA/tasks/WUyDAwy3Rp6oQcuaUzgvZw/details [3]: https://tools.taskcluster.net/groups/e5G50QyjQ7uV1PrEqfrdmA/tasks/RuawcF8ISMCykE4iVV-WdQ/details
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → DUPLICATE
jcristau, bug 1425571 is fairly close to done. Would you like email only for failures, or for successful tasks too ? Fair warning - each release graph has about 4500 tasks so that's a lot of email in the second case!
Flags: needinfo?(jcristau)
I don't know about other consumers of that mailing list, but I definitely don't need to get notifications for the successful tasks.
Flags: needinfo?(jcristau)
You need to log in before you can comment on or make changes to this bug.