See: https://sentry.io/taskcluster/taskcluster-queue/issues/205875529/ This caused 5min downtime, and we fixed by restarting the queue. Long term solution: Fix pulse-publisher We probably upgraded the underlying AMQP library and didn't check semantics changes for emit 'error'. Or something like that. Or maybe tc-monitor failed to crash the process. TODO: Investigate why and when errors are emitted.
Looks like forced connection shutdown messages now issue a 'close' event, see: https://github.com/squaremo/amqp.node/issues/110 Hence, we didn't crash and by implication didn't reconnect.
This has been merged.