1317729 - Shutting down VPN tunnels for Plan B Bugzilla account

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps: Other, task)

Description

[Richard Weiss [:r2]]

We are closing down the Plan B Bugzilla account and will be shutting down the VPN tunnels for the account.

Comment 1

[Peter Radcliffe [:pir]]

This VPN tunnel?

18:02 <@nagios-scl3:#sysadmins> (IRC) Tue 10:02:39 PST [5277] 
  fw1.ops.scl3.mozilla.net:BGP usw2 vpn-8a302f98-2 is CRITICAL: SNMP CRITICAL - 
  BGP sess vpn-8a302f98-2 (usw2/169.254.13.193) connection state *2* . 
  (http://m.mozilla.org/BGP+usw2+vpn-8a302f98-2)

18:05 <@nagios-scl3:#sysadmins> (IRC) Tue 10:05:11 PST [5283] 
  fw1.ops.scl3.mozilla.net:BGP usw2 vpn-8b302f99-1 is CRITICAL: SNMP CRITICAL - 
  BGP sess vpn-8b302f99-1 (usw2/169.254.13.161) connection state *2* . 
  (http://m.mozilla.org/BGP+usw2+vpn-8b302f99-1)

Comment 2

[Peter Radcliffe [:pir]]

18:25 <@nagios-scl3:#sysadmins> (IRC) Tue 10:25:48 PST [5326] 
  fw1.ops.scl3.mozilla.net:BGP usw2 vpn-8b302f99-1 is CRITICAL: SNMP CRITICAL - 
  BGP sess vpn-8b302f99-1 (usw2/169.254.13.161) connection state *3* . 
  (http://m.mozilla.org/BGP+usw2+vpn-8b302f99-1)

18:32 <@nagios-scl3:#sysadmins> (IRC) Tue 10:32:47 PST [5343] 
  fw1.ops.scl3.mozilla.net:BGP usw2 vpn-8a302f98-2 is CRITICAL: SNMP CRITICAL - 
  BGP sess vpn-8a302f98-2 (usw2/169.254.13.193) connection state *2* . 
  (http://m.mozilla.org/BGP+usw2+vpn-8a302f98-2)

Comment 3

[Vinh Hua [:vinh]]

•dcurado> vinh: I can confirm that the 4 BGP sessions that are down on fw1.scl3 (along with their associated IPsec VPNs to AWS) are part of the "Plan B" VPCs

10:33 AM <vinh> thanks dcurado 

10:33 AM <•dcurado> According to the documentation I have kept on these VPNs, there are more VPNs that I suppose may be taken down.

10:34 AM <•dcurado> I see 4 pairs of BGP/IPSec VPNs for Plan B

Comment 4

[Dave Curado :dcurado]

I will remove these retired VPNs shortly, to stop the noise in netops-alerts and anywhere else.

Comment 5

[Peter Radcliffe [:pir]]

:r2 :

These decoms HAVE to go to netops and the MOC *BEFORE* they get turned down on your end. Otherwise the first thing we hear about them is nagios alerts. This wastes time for the oncall people in both teams trying to work out if it is an actual outage or a decom, working out which account the VPNs are related to and increases alert fatigue.

Don't shut VPNs down before the monitoring is confirmed to have been removed.

Comment 6

[Vinh Hua [:vinh]]

Removed nagios monitoring in commit 6deb5d84723cbc2781499f513b92d6e16270d152

Comment 7

[Vinh Hua [:vinh]]

Attachment: nagios check

Comment 8

[Dave Curado :dcurado]

OK, I removed the VPNs that were down.
There are still more "plan B account" VPNs, which I am guessing will be taken down.

Apologies if I took anything down too soon.
I guess the order should be:

MOC stops monitoring them.
netops deletes them.
AWS folks delete their side.

Richard, please let me know if/when the other VPNs will be taken down?
Thanks.

Comment 9

[Richard Weiss [:r2]]

Dave, there are no other active Plan B accounts. Can you tell me what your inventory shows?

Comment 10

[Dave Curado :dcurado]

Hi Richard,

Right, just one "Plan B" account, but more than one VPC in that account, and each VPC with a VPN.
Specifically:

Plan B us-east-1 "stage" 
Plan B us-east-1 "production" 
Plan B us-west-2 "stage"
Plan B us-west-2 "production"

So far we have seen the VPNs to the VPCs in us-west-2 go down, but the VPNs to both us-east-1 VPCs
are still up and running.  

HTHs, 
Dave

Comment 11

[Peter Radcliffe [:pir]]

(In reply to Dave Curado :dcurado from comment #8)
> Apologies if I took anything down too soon.

That comment was directed at :r2, not you, Dave.

> I guess the order should be:
> 
> MOC stops monitoring them.
> netops deletes them.
> AWS folks delete their side.

That would be ideal, yes, thanks.

Comment 12

[Dave Curado :dcurado]

In the future, before shutting any of AWS VPNs down, (assuming they are still up) I'll ask the
MOC to remove the monitoring first.

Thanks

Comment 13

[Dave Williams [:daveio]]

Further shutdowns - see bug 1317978

2016-11-16 14:30:42     @justdave       daveio: see bug 1317729
2016-11-16 14:31:45     daveio  justdave: are you saying that further tunnels have been taken down without a request to moc to remove monitoring?
2016-11-16 14:32:15     @justdave       daveio: yes, it appears that way

Comment 14

[Dave Williams [:daveio]]

I'll remove monitoring for these further VPNs now.

Comment 16

[Dave Williams [:daveio]]

Removed in commit 1359d2859b2048d253e8988bf1c839cbcbde0cff

Comment 17

[Dave Curado :dcurado]

OK, as the monitoring has been shut down, I'll remove the VPNs this morning.

Comment 18

[Dave Curado :dcurado]

VPNs have been deleted.
Closing this bug.  
Re-open if there is any further action need.
Thanks.