Nagios is going off about a bunch of things, I can't connect to mpt-vpn in any way, and we're starting to get alarms about build machines.
Things appear back now...lowering sev
Assigning to dmoore for investigation. Emailed him with some output already. Short story: Primary FWSM failed and the Standby never fully finished the takeover until I rebooted the Primary (core1). Looks like it was about 13 mins of outage. For RelEng, would have affected any inter-vlan traffic (Vlan90 to Vlan71).
This was a failover failure. The failover sync process (doorbell_poll) crashed on the Active module, which meant it began sending incomplete health messages. The two modules could not successfully negotiate a failover, leaving both of them in an intermediate state. There is no pending fix from Cisco.