In order to properly restore all BGP sessions on border1, it is necessary to establish them in a particular order to avoid overloading the CPU. In particular, the overhead of establishing a BGP session with Mzima can cause timeouts and flapping of our OSPF sessions with GNI. If our BGP session with GNI is already established, this can cause an unintentional reset which quickly leads to a fatal chain-reaction of BGP session failures.
It appears the desired bring-up order will be:
* Any2 peers
* Mzima #1
* Mzima #2
* Internal sessions
Currently, only our session with GNI is active on border1. To bring all peers online, we will need to intentionally terminate this session and reactivate all peers in the above order.
Note: Ideally, the OSPF issue will be resolved by experimenting with more liberal timeouts during a future maintenance window.
During the phased activation of BGP peers, CPU utilization remained higher than expected. border1 was switching a significant number (~75%) of packets in software, saturating the CPU. Consultation with Cisco TAC suggested a memory management problem with the TCAM. The quickest solution was a reboot of border1. After the reboot, all BGP peers were restored without issue and CPU utilization remained minimal.
A second Level3 gigabit circuit, previously disabled due to faulty media, was also enabled during this window.