707897 - MTV1 unreachable on the network

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Ashish Vijayaram [:ashish]]

Beginning 00:47 Pacific, hosts in MV/MTV1 paged for network connectivity. Ganglia, build hosts, KVM and office infrastructure is down.

Comment 1

[Ashish Vijayaram [:ashish]]

Recoveries at 01:18.

Comment 2

[Aaron]

AboveNet went down.  I called and they said they were doing maint.  Their maint is complete and the circuit has recovered.  

They notified us by email, but of course we don't currently have access to said email.  Here is a copy I had them send to my personal email address:

Announcement: Emergency Maintenance: SJC2 12/06/2011, 00:01 - 04:00 PST

Message:
Dear AboveNet Customer,

AboveNet is sending this announcement to all technical contacts who are impacted by this emergency hardware maintenance.  The details of this maintenance are below.


Date & Time:  12/06/2011, 00:01 - 04:00 PST

Location(s):  SJC2

Description:  AboveNet Engineers will replace hardware that is in jeopardy of failing.

Customer Impact:  Dual attached customers and switch attached customers may experience momentary degraded performance.  Single attached customers should anticipate as much as 15 to 30 minutes of downtime.

Comment 3

[Ravi Pina [:ravi]]

There were 2 old routes on core2.sjc1 which blocked natural routing from doing it's thing.

Pulled the routes and simulated a failover with expected (uninterrupted) results.