Last Comment Bug 481755 - Terminate all BGP sessions on border1 and restore in proper order
: Terminate all BGP sessions on border1 and restore in proper order
Status: RESOLVED FIXED
:
Product: mozilla.org Graveyard
Classification: Graveyard
Component: Server Operations (show other bugs)
: other
: Other Other
: -- normal (vote)
: ---
Assigned To: Derek Moore [:dmoore]
: matthew zeier [:mrz]
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-03-05 16:02 PST by Derek Moore [:dmoore]
Modified: 2015-03-12 08:17 PDT (History)
2 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Derek Moore [:dmoore] 2009-03-05 16:02:10 PST
In order to properly restore all BGP sessions on border1, it is necessary to establish them in a particular order to avoid overloading the CPU. In particular, the overhead of establishing a BGP session with Mzima can cause timeouts and flapping of our OSPF sessions with GNI. If our BGP session with GNI is already established, this can cause an unintentional reset which quickly leads to a fatal chain-reaction of BGP session failures.

It appears the desired bring-up order will be:

* Any2 peers
* Mzima #1
* Mzima #2
* Internal sessions
* GNI

Currently, only our session with GNI is active on border1. To bring all peers online, we will need to intentionally terminate this session and reactivate all peers in the above order.
Comment 1 Derek Moore [:dmoore] 2009-03-05 16:03:25 PST
Note: Ideally, the OSPF issue will be resolved by experimenting with more liberal timeouts during a future maintenance window.
Comment 2 Derek Moore [:dmoore] 2009-03-06 04:14:45 PST
Process completed.

During the phased activation of BGP peers, CPU utilization remained higher than expected. border1 was switching a significant number (~75%) of packets in software, saturating the CPU. Consultation with Cisco TAC suggested a memory management problem with the TCAM. The quickest solution was a reboot of border1. After the reboot, all BGP peers were restored without issue and CPU utilization remained minimal.

A second Level3 gigabit circuit, previously disabled due to faulty media, was also enabled during this window.

Note You need to log in before you can comment on or make changes to this bug.