Closed
Bug 985164
Opened 11 years ago
Closed 11 years ago
Zayo Circuit Maintenance IMPACT: PHX
Categories
(Infrastructure & Operations Graveyard :: NetOps: DC Carrier, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: jbarnell, Assigned: dcurado)
Details
Zayo is performing maintenance on a line card serving PHX. This maintenance should resolve the alarms on the carrier equipment. NetOps is working with the carrier to obtain a MOP and ensure linecard diversity at Zayo.
Maintenance Ticket Details:
aintenance Ticket #: TTN-414552
Urgency: Planned
Maintenance Window: 00:01 - 01:30 Mountain
Primary Date: 29-Mar 2014
Backup Date: 30-Mar 2014
Location of Maintenance: Phoenix AZ
Reason for Maintenance: Zayo will reseat card to clear alarms on mpr3.phx2.us.
Expected Impact: SA (Service Affecting) Down up to 30 Minutes
Service ID: T00539-00
Reporter | ||
Updated•11 years ago
|
Flags: cab-review?
Assignee | ||
Updated•11 years ago
|
Assignee: network-operations → dcurado
Assignee | ||
Updated•11 years ago
|
Status: NEW → ASSIGNED
Assignee | ||
Comment 1•11 years ago
|
||
I have contacted Donna Turner, our Zayo account representative, to see where we are in the
process of having one of our two circuits to Phoenix re-provisioned (or redesigned) so that
it does not go through the same equipment in Phoenix.
Donna states that they are working on it.
I have pointed out to Donna that Zayo will be doing maintenance 10 days from now which, if
one of our circuits is not reprovisioned for diversity, will cause a repeat of their last
maintenance window. I have pointed out to Donna that we tried to work on the redesign by
following their suggestion of opening a ticket with their NOC, and that their NOC gave a
not-so-customer-friendly response a few days later. ("we don't design things, we fix broken
things") Time was wasted there.
I have asked Donna to escalate this issue with Zayo.
To be clear, if Zayo does not get one of our two circuits moved, Mozilla's network will
survive alright -- just like the last time. Traffic from the Internet to our data centers
will continue to work correctly. The problem will be our internal VPN, connections over
that VPN (the 10.x.x.x/8 network) to Phoenix will not work during their maintenance window.
While I will continue to push with Zayo, we should keep in mind that we don't have
visibility into Zayo's infrastructure. For all we know, they may have limited capacity
in Phoenix, and have no way to diversify our circuit. I will continue to update this
bug with any new information as I receive it.
Assignee | ||
Comment 2•11 years ago
|
||
Donna from Zayo just called me, here's the story...
1. when these circuits were sold to Mozilla, they were from AboveNet.
2. at that time, AboveNet only had one "router" in Phoenix, and according to
them, they explained that to Mozilla. i.e. we bought these circuits knowing
they were not diverse.
3. Zayo then purchased AboveNet, and also installed more equipment in Phoenix.
4. They can move one of our circuits to a different piece of equipment in Phoenix, but
they are saying there are costs involved. They want to charge us $30,000 to make
this change.
5. Donna realized we won't be thrilled about that, and put together a compromise:
- that we extend our term for these circuits for another 18 months
- Specifically, we signed a 3 year term that will be up on 10/9/2015.
- What she is proposing is that we end that contract right away, and start
a new 3 year contract, which of course will take us to March 2017.
- There will be no change in price for the circuits.
6. If we want to get this done before the planned maintenance on 3/29/2014, we need
to pull the trigger on this ASAP.
I will write to James and Albert immediately so this can be decided by upper management.
Comment 3•11 years ago
|
||
This was communicated to the MOC during the CAB of Mar 19. They will take
follow up actions as appropriate with further information from netops.
Flags: cab-review? → cab-review+
Assignee | ||
Comment 4•11 years ago
|
||
There is a meeting between Mozilla and Zayo on Friday March 21.
Albert indicated he'd like to cover this issue/decision with Zayo and Brian Evars at that time.
Donna has sent us a PDF of the new contract, I have forwarded that on to management.
Assignee | ||
Comment 5•11 years ago
|
||
The circuits to PHX1 did not go down during Zayo's maint. window, perhaps they pushed it
off by 24 hours. Their email regarding this maintenance said that was possible.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Updated•9 years ago
|
Change Request: --- → approved
Flags: cab-review+
Updated•2 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•