1294679 - Release Automation (Promotion) notifications delayed several hours.

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Justin Wood (:Callek)]

So for 49.0b3 overnight, we had several notifications stall (no e-mail)...

Which, while I was away from home led me to initially believe we had a problem with the release.

When I realized we were not actually broken on the release, I went back away without giving it another thought for the night, suspecting Google Groups was slow or rate-limiting us...

Today on further inspection I saw that the IRC channel for the notifications had a near-exact similar delay.

One such task:
https://tools.taskcluster.net/task-inspector/#kzcjJomaTaaWMKNlKMtxpg

Run 0 Finished about 15 hours ago (from the time of this bug) however the e-mail for that run came only about 4 hours ago...

Our gap in notification (Timestamps ET):
* 4:48 PM yesterday
   * https://tools.taskcluster.net/task-inspector/#nkzRp5iHSQePemD32ypA1A
   * Finished ~ 17 hours ago [[2016-08-11 20:48:06.718Z]]
* 4:10 AM today
   * https://tools.taskcluster.net/task-inspector/#KAWB7FKsQbymBSuMUcyCfg
   * Finished ~ 17 hours ago [[2016-08-11 20:48:59.055Z]]

This was a total of 465 e-mails going in over the course of minutes this AM. for all these tasks that completed overnight.

Comment 1

[Justin Wood (:Callek)]

9:16 AM <•Callek> rail: in our time, 5:45p is when the dot release started, and about 6am is when our *first* e-mail started coming
9:16 AM <•Callek> and pretty much all emails came at around the same time
9:16 AM <•rail> probably the daemon doesn't survive IRC kicks
9:16 AM <•Callek> rail: email and IRC

Comment 2

[Rail Aliiev [:rail]]

I think this is fixed now. The bot has been very stable for weeks