downtime request for scl1 intra-DC fixes

RESOLVED FIXED

Status

RESOLVED FIXED
7 years ago
6 years ago

People

(Reporter: dustin, Assigned: dustin)

Tracking

Details

As discussed in the relops meeting, we'd like to plan at least one tree closure to work on the links in scl1.  The details of what is to be done are not fully worked out yet, but the general outline is:

 - add new interfaces on the SRX650's
 - switch some traffic over to them
 - switch the primary uplink to P2P

with testing and verification along the way.  It's likely we could do most of this without a tree closure, but if the trees are closed anyway, we'll do disruptive testing.

We may want a second tree closure one or more days later to do the P2P change, although we will test its failover during the first downtime.

For the moment, let's start by scheduling one downtime.  Parameters I know of (please correct if these are wrong):
 - 3h
 - not during a release

The pacific evening of November 10th was mooted as a possibility, although ravi is in NZ that week, so that may make the timing a little funny.  Discuss.  Negotiate.
Flags: needs-treeclosure?
colo-trip: --- → scl1
16:58 < dustin> thurs at 6pm pacific = fri at 3 pm auckland -- work for you?
16:59 < dustin> or thereabouts
16:59 < ravi> yep
16:59 < dustin> kk
Flags: needs-treeclosure? → needs-treeclosure+
Coop ok'd the above time (6pm pacific) via email/IRC.
downtime was completed with very minor blips, more detailed look at the logs needs to happen to know for sure.

calling this a win - dustin to file a tl;dr report along with a longer detailed one later

thanks Rel|Net Ops!
Flags: needs-treeclosure+
ick, forgot to record the duration:

trees closed at 1800 PST and reopened at 2130 PST (overage due to bear being extra cautious)
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.