release-automation-notifications doesn't get email on failures in 59



Release Engineering
Release Automation
a month ago
10 days ago


(Reporter: jcristau, Assigned: mtabara)


Firefox Tracking Flags

(Not tracked)




a month ago
In 58 the release-automation-notifications list got email for every completed and failed release task.  (For a while it was only failed tasks, but that got backed out according to bug 1412014.)

For 59.0b1 no emails seem to be sent, even though we had plenty of builds, and plenty of failed tasks.

Comment 1

a month ago
Oddly enough, one of them at least arrived, some time later on: "FAILED: [devedition 59.0b1 build11/mozilla-beta] release-version-bump-devedition task"

Comment 2

a month ago
59.0b2 was full green graph so no emails arrived -
Pushing/Shipping graph were as green as `promote` one.

I'll debug now the 59.0b1 where the original issues occurred.


a month ago
Assignee: nobody → mtabara

Comment 3

a month ago
First, a little bit of context.

General context:
* TC jobs are indexed under some certain routes
* there's certain filters on those specific routes to trigger pulse messages
* pulse-notify (aka release-notifications) handles processes thoses notifications and sends them up

a) reads pulse messages via the pulseguardian account and processes them
b) sends the notification over to IRC
c) sends AWS SNS messages in the cloud for emails

So in order for this to function properly we need:
i) correct task defined with routes
ii) filtering setup to trigger pulse messages upon those routes
iii) release-notifications service up and running in Heroku to read those pulse messages via pulseguardian
iv) process and send the notifications in irc/email via AWS SES

Comment 4

a month ago
I started debugging from latter to former and didn't notice anything suspicious in the pulse-notify serivce in Heroku. Everything seems fine there, as well as the pulseguardian. I then checked the task definitions to make sure the email addresses are correct by-project and the notifications logic is there. All good.

Then I noticed the lack of routes in some particular tasks.
We've had 11 different builds for Devedition-59.0b1 of which the majority of the failing tasks were:
* l10n jobs
* update verification
* version bump

Looking close at these tasks, I noticed that these tasks were precisely the ones missing the routes: 
E.g. nightly-l10n failing, no routes defined [1]
E.g. update verification failing, no routes defined [2]
E.g. version bump failing, emails sent because routes present [3]



a month ago
Last Resolved: a month ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1425571
jcristau, bug 1425571 is fairly close to done. Would you like email only for failures, or for successful tasks too ? Fair warning - each release graph has about 4500 tasks so that's a lot of email in the second case!
Flags: needinfo?(jcristau)

Comment 7

10 days ago
I don't know about other consumers of that mailing list, but I definitely don't need to get notifications for the successful tasks.
Flags: needinfo?(jcristau)
You need to log in before you can comment on or make changes to this bug.