1562289 - Allocate and set up new IP space for releng tier3 workers in mdc1 and mdc2

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps: DC Other, task)

Description

[Jake Watkins [:dividehex]]

As discussed with secops during the June 2019 All Hands, we will need to carve out network space for tier3 taskcluster workers which carry a greater security risk and impact than the talos/try taskcluster workers located in the test and wintest releng networks. This work currently block setup and implementation of mac minis PGO build tasks. See bug 1395325 and bug 1530732

Please allocate and setup a new vlan and network segment for tier3 related workers in mdc1 and mdc2.

This new network segment should be within the releng mdc1/2 cidr ranges and /22 bits wide. This network should be similar to the test.releng.mdc[1,2].mozilla.com networks with regards to firewall policies.

Releng cidr ranges in mdc1 and mdc2:
10.49.0.0 mdc1
10.51.0.0 mdc2

Domains:
.tier3.releng.mdc1.mozilla.com
.tier3.releng.mdc2.mozilla.com

Comment 1

[Nadia]

Looks like we have the following vlans free:
MDC 1 10.49.60.0/22 range 10.49.60.1 - 10.49.63.254 VLAN 260
MDC 2 10.51.60.0/22 range 10.51.60.1 - 10.51.63.254 VLAN 260

getting JohnB's ok to use the ranges before I proceed.

Comment 2

[Nadia]

got the go ahead to build the above subnets and vlans for releng from John.

Comment 3

[Nadia]

Created Layer 2 vlan and name: vlan 260 tier3.releng.mdc1 in all mdc1 switches
Created Layer 2 vlan and name: vlan 260 tier3.releng.mdc2 in all mdc2 switches

Comment 4

[Nadia]

Created a layer 3 SVI on the Palo Alto firewall tagged vlan 260
MDC 1 10.49.60.0/22
MDC 2 10.51.60.0/22

Created a zone called tier3-releng and enabled dhcp

Comment 5

[Nadia]

Added the new tier3-releng zone to the following rules:
Rule 183 anything-to-git-zlb
Rule 226 bacula-bidirectional
Rule 215 infoblox-dhcp-client
Rule 227 infoblox-dhcp-server
Rule 228 infoblox-udp67
Rule 212 Key Management Server - MS
Rule 170 Proxy ByPass

If you want me to remove the zone from any of the rules let me know, also if I missed any rules let me know as well.

Comment 6

[Nadia]

Created both MDC1 and MDC2 subnets in infoblox.

Comment 7

[Nadia]

(In reply to Jake Watkins [:dividehex] from comment #0)

As discussed with secops during the June 2019 All Hands, we will need to carve out network space for tier3 taskcluster workers which carry a greater security risk and impact than the talos/try taskcluster workers located in the test and wintest releng networks. This work currently block setup and implementation of mac minis PGO build tasks. See bug 1395325 and bug 1530732

Please allocate and setup a new vlan and network segment for tier3 related workers in mdc1 and mdc2.

This new network segment should be within the releng mdc1/2 cidr ranges and /22 bits wide. This network should be similar to the test.releng.mdc[1,2].mozilla.com networks with regards to firewall policies.

Releng cidr ranges in mdc1 and mdc2:
10.49.0.0 mdc1
10.51.0.0 mdc2

Domains:
.tier3.releng.mdc1.mozilla.com
.tier3.releng.mdc2.mozilla.com

OK this should be good to test and verify.

If this is working completely to your liking please close this out.

Regards,
Nadia

Comment 8

[Chris Knowles [:cknowles]]

Can we please trunk this VLAN into the UCS in MDC1 and MDC2?

Comment 9

[Nadia]

access05.private.mdc1#sh run int po17
interface Port-Channel17
description esxucs1-A
switchport trunk allowed vlan 5,8,60,62,69-72,74-75,77,81,86-87,121,124-125,130,220,240,248,256,260,275,278,3000-3002
switchport mode trunk
mlag 17
access05.private.mdc1#sh run int po18
interface Port-Channel18
description esxucs1-B
switchport trunk allowed vlan 5,8,60,62,69-72,74-75,77,81,86-87,121,124-125,130,220,240,248,256,260,275,278,3000-3002
switchport mode trunk
mlag 18

access06.private.mdc1#sh run int po17
interface Port-Channel17
description esxucs1-A
switchport trunk allowed vlan 5,8,60,62,69-72,74-75,77,81,86-87,121,124-125,130,220,240,248,256,260,275,278,3000-3002
switchport mode trunk
mlag 17
access06.private.mdc1#sh run int po18
interface Port-Channel18
description esxucs1-B
switchport trunk allowed vlan 5,8,60,62,69-72,74-75,77,81,86-87,121,124-125,130,220,240,248,256,260,275,278,3000-3002
switchport mode trunk
mlag 18

Comment 10

[Chris Knowles [:cknowles]]

Excellent - But I'm still not able to DHCP addresses in that VLAN from the UCS. I've confirmed that the UCS/ESX are setup to handle VLAN260 traffic, but when I try to DHCP, I don't see any logging in the infoblox to indicate that things are getting there.

Any help is appreciated.

Comment 11

[Jake Watkins [:dividehex]]

(In reply to Chris Knowles [:cknowles] from comment #10)

Excellent - But I'm still not able to DHCP addresses in that VLAN from the UCS. I've confirmed that the UCS/ESX are setup to handle VLAN260 traffic, but when I try to DHCP, I don't see any logging in the infoblox to indicate that things are getting there.

Any help is appreciated.

Was the dhcphelper setup to forward requests to infoblox on the new vlans?

Comment 12

[Nadia]

Added vlan 260 to vxlan, core, and firewall uplinks verified with Chris over slack now gets dhcp requests.