814859 - VPN connections to Amazon VPCs are down

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Rail Aliiev [:rail]]

It looks like VPN connections in all used Amazon regions are down at the moment. As a result no routing from SCL3 to AWS. Filing as a blocker because the trees are closed, sorry.

Comment 1

[Shyam Mani [:fox2mike]]

Can you provide additional steps to verify? Or what you tried to come to this conclusion?

Comment 2

[Shyam Mani [:fox2mike]]

Arzhel was online and already poking. He paged Casey who's now looking into the issue.

Comment 3

[casey ransom [:casey]]

I'm unsure as to cause (bug, it appears), but all the vpn connections on fw1.releng.scl3 were no longer passing traffic.  I cleared SA's to try and rebuild the connections but ultimately ended up disabling the VPNs and re-enabling the config which brought them back up.  I've got logs from the event and will scrounge around on juniper's KB to see if there is a known PR for this behavior.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Thanks Casey, jobs are running again. I've reopened the trees.

Comment 5

[Rail Aliiev [:rail]]

Sorry, I've been offline to help with debugging the problem. The VPN connections look OK now. Not a blocker anymore.

Comment 6

[casey ransom [:casey]]

I wasn't able to track down a specific PR related to our specific errors however there were a number of ipsec bug fixes and enhancements between 11.1 and 11.4.  Our goal is to update this and other remaining firewalls to 11.4 during a window that we are trying to schedule for 12/16.  Making this bug dependent on that work.