Closed Bug 985164 Opened 11 years ago Closed 11 years ago

Zayo Circuit Maintenance IMPACT: PHX

Categories

(Infrastructure & Operations Graveyard :: NetOps: DC Carrier, task)

Other
Other
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jbarnell, Assigned: dcurado)

Details

Zayo is performing maintenance on a line card serving PHX. This maintenance should resolve the alarms on the carrier equipment. NetOps is working with the carrier to obtain a MOP and ensure linecard diversity at Zayo. Maintenance Ticket Details: aintenance Ticket #: TTN-414552 Urgency: Planned Maintenance Window: 00:01 - 01:30 Mountain Primary Date: 29-Mar 2014 Backup Date: 30-Mar 2014 Location of Maintenance: Phoenix AZ Reason for Maintenance: Zayo will reseat card to clear alarms on mpr3.phx2.us. Expected Impact: SA (Service Affecting) Down up to 30 Minutes Service ID: T00539-00
Flags: cab-review?
Assignee: network-operations → dcurado
Status: NEW → ASSIGNED
I have contacted Donna Turner, our Zayo account representative, to see where we are in the process of having one of our two circuits to Phoenix re-provisioned (or redesigned) so that it does not go through the same equipment in Phoenix. Donna states that they are working on it. I have pointed out to Donna that Zayo will be doing maintenance 10 days from now which, if one of our circuits is not reprovisioned for diversity, will cause a repeat of their last maintenance window. I have pointed out to Donna that we tried to work on the redesign by following their suggestion of opening a ticket with their NOC, and that their NOC gave a not-so-customer-friendly response a few days later. ("we don't design things, we fix broken things") Time was wasted there. I have asked Donna to escalate this issue with Zayo. To be clear, if Zayo does not get one of our two circuits moved, Mozilla's network will survive alright -- just like the last time. Traffic from the Internet to our data centers will continue to work correctly. The problem will be our internal VPN, connections over that VPN (the 10.x.x.x/8 network) to Phoenix will not work during their maintenance window. While I will continue to push with Zayo, we should keep in mind that we don't have visibility into Zayo's infrastructure. For all we know, they may have limited capacity in Phoenix, and have no way to diversify our circuit. I will continue to update this bug with any new information as I receive it.
Donna from Zayo just called me, here's the story... 1. when these circuits were sold to Mozilla, they were from AboveNet. 2. at that time, AboveNet only had one "router" in Phoenix, and according to them, they explained that to Mozilla. i.e. we bought these circuits knowing they were not diverse. 3. Zayo then purchased AboveNet, and also installed more equipment in Phoenix. 4. They can move one of our circuits to a different piece of equipment in Phoenix, but they are saying there are costs involved. They want to charge us $30,000 to make this change. 5. Donna realized we won't be thrilled about that, and put together a compromise: - that we extend our term for these circuits for another 18 months - Specifically, we signed a 3 year term that will be up on 10/9/2015. - What she is proposing is that we end that contract right away, and start a new 3 year contract, which of course will take us to March 2017. - There will be no change in price for the circuits. 6. If we want to get this done before the planned maintenance on 3/29/2014, we need to pull the trigger on this ASAP. I will write to James and Albert immediately so this can be decided by upper management.
This was communicated to the MOC during the CAB of Mar 19. They will take follow up actions as appropriate with further information from netops.
Flags: cab-review? → cab-review+
There is a meeting between Mozilla and Zayo on Friday March 21. Albert indicated he'd like to cover this issue/decision with Zayo and Brian Evars at that time. Donna has sent us a PDF of the new contract, I have forwarded that on to management.
The circuits to PHX1 did not go down during Zayo's maint. window, perhaps they pushed it off by 24 hours. Their email regarding this maintenance said that was possible.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Change Request: --- → approved
Flags: cab-review+
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.