All trees closed due to Taskcluster 503 errors burning all builds

RESOLVED FIXED

Status

--
blocker
RESOLVED FIXED
3 years ago
6 months ago

People

(Reporter: RyanVM, Unassigned)

Tracking

Details

Comment hidden (empty)

Comment 1

3 years ago
queue was restarted at around 17:07Z and errors stopped.  So far Jonas has noticed issues with communication between various taskcluster components and Pulse starting around 16:23Z.
From consulting :cyliang, looks like we've got a network issue between AWS and PHX1, dropping packets at a high % inside Telia routing space.  New Relic traffic is another service also affected.
At 10:15am PT, netops turned off the circuit to that provide to stop the packet hemorrhage.
(Reporter)

Comment 4

3 years ago
Bustage has been cleared, trees reopened.
(Reporter)

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED

Updated

6 months ago
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.