Terminate all BGP sessions on border1 and restore in proper order

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
9 years ago
2 years ago

People

(Reporter: dmoore, Assigned: dmoore)

Tracking

Details

(Assignee)

Description

9 years ago
In order to properly restore all BGP sessions on border1, it is necessary to establish them in a particular order to avoid overloading the CPU. In particular, the overhead of establishing a BGP session with Mzima can cause timeouts and flapping of our OSPF sessions with GNI. If our BGP session with GNI is already established, this can cause an unintentional reset which quickly leads to a fatal chain-reaction of BGP session failures.

It appears the desired bring-up order will be:

* Any2 peers
* Mzima #1
* Mzima #2
* Internal sessions
* GNI

Currently, only our session with GNI is active on border1. To bring all peers online, we will need to intentionally terminate this session and reactivate all peers in the above order.
(Assignee)

Comment 1

9 years ago
Note: Ideally, the OSPF issue will be resolved by experimenting with more liberal timeouts during a future maintenance window.

Updated

9 years ago
Group: infra
(Assignee)

Comment 2

9 years ago
Process completed.

During the phased activation of BGP peers, CPU utilization remained higher than expected. border1 was switching a significant number (~75%) of packets in software, saturating the CPU. Consultation with Cisco TAC suggested a memory management problem with the TCAM. The quickest solution was a reboot of border1. After the reboot, all BGP peers were restored without issue and CPU utilization remained minimal.

A second Level3 gigabit circuit, previously disabled due to faulty media, was also enabled during this window.
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.