Release Automation (Promotion) notifications delayed several hours.

RESOLVED FIXED

Status

Release Engineering
Release Automation: Other
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: Callek, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
So for 49.0b3 overnight, we had several notifications stall (no e-mail)...

Which, while I was away from home led me to initially believe we had a problem with the release.

When I realized we were not actually broken on the release, I went back away without giving it another thought for the night, suspecting Google Groups was slow or rate-limiting us...

Today on further inspection I saw that the IRC channel for the notifications had a near-exact similar delay.

One such task:
https://tools.taskcluster.net/task-inspector/#kzcjJomaTaaWMKNlKMtxpg

Run 0 Finished about 15 hours ago (from the time of this bug) however the e-mail for that run came only about 4 hours ago...

Our gap in notification (Timestamps ET):
* 4:48 PM yesterday
   * https://tools.taskcluster.net/task-inspector/#nkzRp5iHSQePemD32ypA1A
   * Finished ~ 17 hours ago [[2016-08-11 20:48:06.718Z]]
* 4:10 AM today
   * https://tools.taskcluster.net/task-inspector/#KAWB7FKsQbymBSuMUcyCfg
   * Finished ~ 17 hours ago [[2016-08-11 20:48:59.055Z]]

This was a total of 465 e-mails going in over the course of minutes this AM. for all these tasks that completed overnight.
(Reporter)

Updated

2 years ago
Depends on: 1294425
(Reporter)

Comment 1

2 years ago
9:16 AM <•Callek> rail: in our time, 5:45p is when the dot release started, and about 6am is when our *first* e-mail started coming
9:16 AM <•Callek> and pretty much all emails came at around the same time
9:16 AM <•rail> probably the daemon doesn't survive IRC kicks
9:16 AM <•Callek> rail: email and IRC
I think this is fixed now. The bot has been very stable for weeks
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.