Closed Bug 697576 Opened 14 years ago Closed 14 years ago

short scl1 outage while on P2P

Categories

(Infrastructure & Operations :: RelOps: General, task)

Product:
Infrastructure & Operations
Component:
RelOps: General
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Unassigned)

Details

At about 14:18 today, we had an outage on the link from scl1 to sjc1, shortly after switching the primary connection from VPN to P2P. Once the alerts began, ravi switched back to the VPN and they cleared. This packet loss would have severed active connections in and out of scl1. Connections within scl1 should not have been affected. We will file a ticket with Abovenet.
(the link-flip was for bug 697228)
Plan for remediation is now in bug 688506. As the outage is over, this bug is fixed.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
(In reply to Dustin J. Mitchell [:dustin] from comment #2) > Plan for remediation is now in bug 688506. bug#688506 is blocked from us. Please cc' RelEng, or provide a summary here for our education. For example, we would like to know is this outage related to the previous 2 recent scl1 outages tracked in bug#694877? > As the outage is over, this bug > is fixed. How long was the duration of the outage?
(In reply to John O'Duinn [:joduinn] from comment #3) > bug#688506 is blocked from us. Please cc' RelEng, or provide a summary here > for our education. I've cc'd you. The summary of bug 688506 is that the interface-saturation issues we've been seeing for some time may be related to the P2P issues, and the fix may be local (splitting traffic over multiple interfaces on the SRX), rather than with Abovenet. Netops still needs to determine if this is, in fact, the case, and plan a fix accordingly. That work is ongoing, and Amy and I are involved in the conversations. We'll file a bug when a fix is ready, most likely requiring or recommending a tree closure. > For example, we would like to know is this outage related to the previous 2 > recent scl1 outages tracked in bug#694877? It was related in the sense that we saw packet loss on the P2P link some time after switching to it, and brought things back online by switching back to the IPSec link. > How long was the duration of the outage? The nagios alerts went on for a while, but the outage itself was 5-10 minutes.
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.