Closed Bug 1201142 Opened 9 years ago Closed 9 years ago

All trees closed due to Taskcluster 503 errors burning all builds

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

task
Not set
blocker

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: RyanVM, Unassigned)

References

Details

      No description provided.
queue was restarted at around 17:07Z and errors stopped.  So far Jonas has noticed issues with communication between various taskcluster components and Pulse starting around 16:23Z.
From consulting :cyliang, looks like we've got a network issue between AWS and PHX1, dropping packets at a high % inside Telia routing space.  New Relic traffic is another service also affected.
At 10:15am PT, netops turned off the circuit to that provide to stop the packet hemorrhage.
Bustage has been cleared, trees reopened.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.