1042251 - 4 maintenance tasks for netops to perform in the next approved maintenance window

Status: RESOLVED INVALID

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Dave Curado :dcurado]

In order to continue closing out action items from the 200+ point plan to
reach Network Nirvana, (bug 1040891), netops would like to schedule the
following maintenance changes:

1. Finish configuration clean up for PHX1 -- removing temporary configurations
   we had in place on core[12].phx1 and fw1.phx1.  
   Expected impact: none.  (but, let's do this in a maint window anyway)

2. set next-hop-self as an export policy on the ibgp mesh between 
   border1.scl3, border2.scl3, border1.pao1, and border1.sjc2
   Expected impact: some routing churn, but nothing horrible.
    That said, we definitely want to do this in a maint. window
    in case we get surprised by some legacy configuration.
    If that should happen, we'd back out of this change quickly.

3. remove graceful-restart from the following devices:

	+ agg1.s301.ops.phx1.mozilla.net
	+ border1.console.pao1.mozilla.net
	+ border1.console.scl3.mozilla.net
	+ border1.console.sjc2.mozilla.net
	+ border1.phx1.mozilla.net
	+ border2.console.scl3.mozilla.net
	+ border2.phx1.mozilla.net
	+ core1.corp.console.scl3.mozilla.net
	+ core1.corp.phx1.mozilla.net
	+ core1.svc.phx1.mozilla.net
	+ fw1.akl1.mozilla.net
	+ fw1.corp.console.scl3.mozilla.net
	+ fw1.corp.phx1.mozilla.net
	+ fw1.lon1.mozilla.net
	+ fw1.ops.par1.mozilla.net
	+ fw1.ops.pdx1.mozilla.net
	+ fw1.ops.scl1.mozilla.net
	+ fw1.phx1.mozilla.net
	+ fw1.releng.scl3.mozilla.net
	+ fw1.scl3.mozilla.net
	+ fw1.sfo1.mozilla.net
	+ fw1.svc.phx1.mozilla.net
	+ fw1.tor1.mozilla.net
	+ switch1.r101-10.ops.scl3.mozilla.net
	+ switch1.r301-10.ops.scl3.mozilla.net

There is no documentation on the impact of removing graceful-restart from a
switch, router, or firewall configuration.  While graceful-restart is configured
on these devices, it is *not* configured as part of any protocol configuration.
Worst case: protocol adjacencies will be cleared when this configuration line
 is removed.
Best case: nothing will happen when this configuration line is removed.

Either way, we'll do this change one device at a time, making sure that
the network is in a good working state before moving on to the next device.

Total Maintenance Time: 2 hours
Expected Impact: A series of short periods of routing churn
Worst case: setting next-hop-self on border routers in SCL3 and our POPs causes
a routing failure, and we back out quickly.

Comment 1

[Dave Curado :dcurado]

need to figure out what the fourth task is =-)

Comment 2

[Dave Curado :dcurado]

Broke this into 3 separate bugs for easier tracking.