1277611 - Exit IP addresses for releng's systems in SCL3

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Benson Wong [:mostlygeek]]

Backstory: 

Cloudops is moving the firefox update service, (aus4|aus5).mozilla.org, from SCL3 to AWS. We want to IP whitelist access to the update service's admin server to a small set of IPs from SCL3. 

These are the access patterns: 

1. openvpn => scl3 => admin server
2. releng aws => scl3 => admin server
3. releng scl3 => admin server

What IP addresses from SCL3 do we need to whitelist to enable the above scenarios?

Comment 1

[Dave Curado :dcurado]

Answer: it depends.

What is the IP address of the "admin server" ??? That label is loaded, and can mean many things.

For 1:
It depends: will these servers being moved into AWS have an IPSec based VPN connecting them to SCL3?
If so, then OpenVPN users will not have their source IP NAT'd.
If not, then OpenVPN users will have their source IP NAT'd.

For 2: releng aws?  What does that mean?  They have a number of different VPCs in AWS.
      
For 3: define 'admin server'

Rather than speak in general terms like "releng scl3" or "admin server", perhaps use
IP addresses or FQDNs.

Thank you.

Comment 2

[Benson Wong [:mostlygeek]]

Clarify the access patterns: 

1. user ~~ (internet) ~~> openvpn|scl3 ~~ (internet) ~~> admin server
2. releng aws ~~ (ipsec|internet) ~~> scl3 ~~ (internet) ~~> admin server
3. releng scl3 ~~ (internet) ~~> admin server


Re #1 and #3: 
- The Admin Server will have a static, public IP address. 
- Basically a box sitting on the Internet. 
- The servers being moved won't have an IPSec based VPN with SCL3.
- An OpenVPN ACL has been setup to route the static IP through the VPN (Bug 1275722)
- Runs a python web app, https. Has privileged access to the database.


Re #2: 
- releng aws = releng's AWS account. 
- Was not aware they have multiple VPCS. 
- I was told their VPC's default gateway is the IPSEC tunnel to SCL3
- When servers in their VPC access Internet, what IP's are they NAT'd to?


I don't know the architecture / systems that RELENG has in their AWS account and in SCL3. 
Doing some detective work here. 

I have a requirement that their systems in AWS and SCL3 are able to talk to the new admin server, over the internet. We don't want the Admin Server's security group to have a 0.0.0.0/0 rule, so I'm asking for specific IP addresses or ranges to whitelist access. A narrow as possible is preferred.

Comment 3

[Amy Rich [:arr] [:arich]]

Dave: for #2 and #3 I think they want to know what 10.26.0.0/16 gets mapped to when egressing scl3 to the internet. Is that one NAT IP or a range, or..?

Benson: I suspect that you'll want to have releng modify the IGW for their AWS VPC to route traffic to your admin server over the internet, unless there's some reason for us to want to route all that back through the IPSec tunnel to scl3.

I'm not sure what our NAT for the VPN looks like, so I can't help on #1.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Amy Rich [:arr] [:arich] from comment #3)
> Benson: I suspect that you'll want to have releng modify the IGW for their
> AWS VPC to route traffic to your admin server over the internet, unless
> there's some reason for us to want to route all that back through the IPSec
> tunnel to scl3.

It's not a lot of data to carry back to SCL3, egress via releng network's NAT and go out to AWS. To use the IGW we'd probably have to whitelist all of AWS, with detrimental impact on security, since we don't control which public IP each build instance is allocated. 

(In reply to Benson Wong [:mostlygeek] from comment #2)
> Re #2: 
> - releng aws = releng's AWS account. 
> - Was not aware they have multiple VPCS. 
> - I was told their VPC's default gateway is the IPSEC tunnel to SCL3
> - When servers in their VPC access Internet, what IP's are they NAT'd to?

Some examples:
scl3 build machine: [[e-mail] ~]$ curl ip.cow.org      ---> 63.245.214.82
use1 build machine: [[e-mail] ~]$ curl ip.cow.org ---> 63.245.214.82
usw2 build machine: [[e-mail] ~]$ curl ip.cow.org ---> 63.245.214.82

Don't know if that is releng specific or not.

I looked at the routing from the last machine to the staging for the new system (balrog-admin.stage.mozaws.net), it comes back to v-1030.core1.releng.scl3 over the IPsec tunnel, then out through v-1032.border2.scl3, our POPs, and zayo to AWS.

> I have a requirement that their systems in AWS and SCL3 are able to talk to
> the new admin server, over the internet. We don't want the Admin Server's
> security group to have a 0.0.0.0/0 rule, so I'm asking for specific IP
> addresses or ranges to whitelist access. A narrow as possible is preferred.

At the releng router we could ensure that only build slaves have access, not try or testers.

Comment 5

[Dave Curado :dcurado]

OK, for #1:  If there will not be an IPSec VPN to the AWS VPC where this "admin server" will live, then OpenVPN users will appear to come from 63.245.214.137.

For #3: When hosts in releng go out to the Internet, they are source NAT'd to 63.245.208.5 or 63.245.214.5 or 63.245.214.82, depending on *where* inside of releng the packets are coming from.
(Sorry, it's just not as simple as one NAT IP or a range... if you need to know specifically which IP
 the packets will appear to come from, we need to find out more about what resource(s) inside of releng
 we're talking about.

For #2: Oh crimeny, is releng still doing that?  Sending stuff from AWS through SCL3, now to get back to AWS.  <sigh>  Those packets would appear to come from 63.245.214.82.  

Please let me know if this gives you the info you need?  If you create a security group that limits access to these source IPs, and things don't work, please let me know... I'll help figure it out.
In your security group, at least at the beginning, perhaps allow ICMP through from anywhere.
On the SCL3 side, all ICMP is allowed, so if you do the same with your security group, (at least in
the "let's get it working" phase) that will make debugging this easier.  Thanks.

Comment 6

[Benson Wong [:mostlygeek]]

Thanks!

I will whitelist the following IP addresses: 

- 63.245.214.137
- 63.245.208.5
- 63.245.214.5
- 63.245.214.82
- ICMP from 0.0.0.0/0.

I think this is enough for me to get started. Resolving the bug. Will reopen if we run into issues.

Comment 7

[Benson Wong [:mostlygeek]]

For posterity: 

Also had to add: 63.245.214.169/32 to the security group. Had to grab these out of openresty's http logs.