890021 - Widespread twisted disconnects across all trees

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps: Other, task)

Description

[Ryan VanderMeulen [:RyanVM]]

Happening in large quantities across all trees. For example:
https://tbpl.mozilla.org/php/getParsedLog.php?id=24900392&tree=Mozilla-Central

All trees closed as of 13:02 PT.

Comment 1

[[e-mail] (:bhearsum)]

Might be a problem with the AWS link? Not sure how to confirm or reject that theory.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

15:49 < netops1> >> fw1.console.releng.scl3.mozilla.net: %-RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer
                    169.254.249.25 (External AS 7224) changed state from Established to Idle (event
                    RecvNotify)
15:50 < netops1> >> fw1.console.releng.scl3.mozilla.net: %-RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer
                    169.254.249.25 (External AS 7224) changed state from OpenConfirm to Established
                    (event RecvKeepAlive)

which basically means that the Amazon end of the VPC went down and came back up again.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Moving to NetOps to verify that links with AWS are up and working.

Comment 4

[[e-mail] (:bhearsum)]

We already know this was a blip with AWS. We're leaving this open until we're comfortable that there won't be anymore. There's anythin gelse to do here AFAIK:
16:14 <@bhearsum|buildduty> RyanVM: well, it looks like this one is out of our 
                            hands. do you want to wait a bit to see if we have any 
                            more disconnects before declaring this fixed?
16:15 < RyanVM> yes, thanks :)
16:15 <@bhearsum|buildduty> ok
16:15 < RyanVM> thanks for the quick response

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Trees reopened at 13:22 PT.

Comment 6

[casey ransom [:casey]]

this is a normal occurrence and happens regularly over normal operation. there's no further investigation for netops. there's no utility in keeping this bug open.

Comment 7

[Ravi Pina [:ravi]]

This is a painful reminder that we had initially went down the path of VPC for Releng with the understanding that doing so could not and would not close the tree.  Sadly this is no longer the case so it would behoove Releng to begin looking how to prevent a IPSEC tunnel to a 3rd party with a *zero* SLA[1] from closing the tree.

Netops has initiatives to help fortify things on our side, but we can only be as good as the weakest link.

[1] http://aws.amazon.com/vpc/faqs/#Q3

 Q. Does the Amazon VPC VPN Connection have a Service Level Agreement (SLA)?
    Not currently.