481755 - Terminate all BGP sessions on border1 and restore in proper order

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Derek Moore [:dmoore]]

In order to properly restore all BGP sessions on border1, it is necessary to establish them in a particular order to avoid overloading the CPU. In particular, the overhead of establishing a BGP session with Mzima can cause timeouts and flapping of our OSPF sessions with GNI. If our BGP session with GNI is already established, this can cause an unintentional reset which quickly leads to a fatal chain-reaction of BGP session failures.

It appears the desired bring-up order will be:

* Any2 peers
* Mzima #1
* Mzima #2
* Internal sessions
* GNI

Currently, only our session with GNI is active on border1. To bring all peers online, we will need to intentionally terminate this session and reactivate all peers in the above order.

Comment 1

[Derek Moore [:dmoore]]

Note: Ideally, the OSPF issue will be resolved by experimenting with more liberal timeouts during a future maintenance window.

Comment 2

[Derek Moore [:dmoore]]

Process completed.

During the phased activation of BGP peers, CPU utilization remained higher than expected. border1 was switching a significant number (~75%) of packets in software, saturating the CPU. Consultation with Cisco TAC suggested a memory management problem with the TCAM. The quickest solution was a reboot of border1. After the reboot, all BGP peers were restored without issue and CPU utilization remained minimal.

A second Level3 gigabit circuit, previously disabled due to faulty media, was also enabled during this window.