Closed Bug 1070432 Opened 12 years ago Closed 12 years ago

link between mtv2 and sfo down.

Categories

(Infrastructure & Operations :: MOC: Problems, task)

Product:
Infrastructure & Operations
Component:
MOC: Problems
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dgarvey, Unassigned)

Details

The curcuit between mtv2 and sfo went down. Called Zayo and opened up a ticket #540031. ETYX/083475//ZYO Zayo - DIA/VPN to SJC2 Trouble reporting number is 866-236-2824
Alert sequence: First mtv2 alert. Sep 19 16:59:50 <nagios-scl3> Fri 16:59:50 PDT [5043] fw1.private.mtv2.mozilla.net:PING is WARNING: PING WARNING - Pac ket loss = 80%, RTA = 2.48 ms (http://m.mozilla.org/PING) Then the network started to recover. Sep 19 17:01:41 <nagios-scl3> Fri 17:01:41 PDT [5056] fw1.private.mtv2.mozilla.net:PING is OK: PING OK - Packet loss = 0%, RTA = 2.21 ms (http://m.mozilla.org/PING) Not so fast, backdown again. Sep 19 17:43:20 <nagios-scl3> Fri 17:41:18 PDT [5377] fw1.private.mtv2.mozilla.net is DOWN :PING CRITICAL - Packet loss = 100% Lastest recovery. Sep 19 18:16:09 <nagios-scl3> Fri 18:11:57 PDT [5728] fw1.private.mtv2.mozilla.net is UP :PING OK - Packet loss = 0%, RTA = 1.97 ms
Component: MOC: Incidents → MOC: Problems
GENERAL INFORMATION Case Number Customer Ticket Number Customer Circuit Id Subject Description Status Sub Status TTN-0000540297 1046129 null Mozilla Corporation / IPYX/078487//ZYO / Hard Down Mozilla \ IPYX/078487//ZYO \ BGP Fail BGP route removed by customer in error, returned about 10 mins about but not working. Service Restored Monitor LATEST CASE COMMENT Current Comment Worked with Dave w/Mozilla. He was working on 63.245.214.0/23 and for some reason we stopped receiving the route. I found the route suppressed due to damping. I cleared damping on that route. We started to receive the route. Customer confirmed everything is good. Service restoring ticket. @re0.mpr4.sjc7.us> clear bgp damping 63.245.214.0/23 @re0.mpr4.sjc7.us> show route receive-protocol bgp 64.125.170.38 inet.0: 515946 destinations, 2780280 routes (515797 active, 36 holddown, 382 hidden) Restart Complete Prefix Nexthop MED Lclpref AS path * 63.245.214.0/23 64.125.170.38 0 36856 I * 63.245.219.0/24 64.125.170.38 36856 I * 63.245.221.0/24 64.125.170.38 36856 I * 63.245.223.0/24 64.125.170.38 36856 I
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.