Closed Bug 1214357 Opened 10 years ago Closed 10 years ago

Open up flows for plan-b-bugzilla-prod replication back to scl3

Categories

(Infrastructure & Operations Graveyard :: NetOps: DC ACL Request, task)

Product:
Infrastructure & Operations Graveyard
Component:
NetOps: DC ACL Request
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: gozer, Assigned: scabral)

References

Details

Cleaner rework of bug 1204119: Flows for MySQL replication between AWS prod and SCL3: The flows should allow bi-directionnal mysql traffic on tcp/3306 SCL3 Prod DBs: bugzilla5.db.scl3.mozilla.com(10.22.70.100) bugzilla6.db.scl3.mozilla.com(10.22.70.101) bugzilla7.db.scl3.mozilla.com(10.22.70.129) bugzilla8.db.scl3.mozilla.com(10.22.70.102) AWS: us-east-1: ProdPrivateSubnetAZ1Cidr(10.162.13.96/27) ProdPrivateSubnetAZ2Cidr(10.162.13.128/27) ProdPrivateSubnetAZ3Cidr(10.162.13.160/27) us-west-2: ProdPrivateSubnetAZ1Cidr(10.164.13.96/27) ProdPrivateSubnetAZ2Cidr(10.164.13.128/27) ProdPrivateSubnetAZ3Cidr(10.164.13.160/27)
working on this.
Status: NEW → ASSIGNED
This set of security policies should now be in place. Policy: allow-mysql-from-nubis-prod, action-type: permit, State: enabled, Index: 118, Scope Policy: 0 Policy Type: Configured Sequence number: 1 From zone: vpc, To zone: db Source addresses: us-west-2-prod-3: 10.164.13.160/27 us-west-2-prod-2: 10.164.13.128/27 us-west-2-prod-1: 10.164.13.96/27 us-east-1-prod-3: 10.162.13.160/27 us-east-1-prod-2: 10.162.13.128/27 us-east-1-prod-1: 10.162.13.96/27 Destination addresses: bugzilla8.db.scl3: 10.22.70.102/32 bugzilla7.db.scl3: 10.22.70.129/32 bugzilla6.db.scl3: 10.22.70.101/32 bugzilla5.db.scl3: 10.22.70.100/32 Application: mysql IP protocol: tcp, ALG: 0, Inactivity timeout: 1800 Source port range: [0-0] Destination port range: [3306-3306] and Policy: allow-mysql-to-nubis-prod, action-type: permit, State: enabled, Index: 84, Scope Policy: 0 Policy Type: Configured Sequence number: 1 From zone: db, To zone: vpc Source addresses: bugzilla8.db.scl3: 10.22.70.102/32 bugzilla7.db.scl3: 10.22.70.129/32 bugzilla6.db.scl3: 10.22.70.101/32 bugzilla5.db.scl3: 10.22.70.100/32 Destination addresses: us-west-2-prod-3: 10.164.13.160/27 us-west-2-prod-2: 10.164.13.128/27 us-west-2-prod-1: 10.164.13.96/27 us-east-1-prod-3: 10.162.13.160/27 us-east-1-prod-2: 10.162.13.128/27 us-east-1-prod-1: 10.162.13.96/27 Application: mysql IP protocol: tcp, ALG: 0, Inactivity timeout: 1800 Source port range: [0-0] Destination port range: [3306-3306] Per policy TCP Options: SYN check: No, SEQ check: No Diff: dcurado@fw1.ops.scl3.mozilla.net> show configuration | compare rollback 1 | no-more [edit security policies from-zone db to-zone vpc] + /* 1214357 */ + policy allow-mysql-to-nubis-prod { + match { + source-address [ bugzilla5.db.scl3 bugzilla6.db.scl3 bugzilla7.db.scl3 bugzilla8.db.scl3 ]; + destination-address planb-prod; + application mysql; + } + then { + permit; + } + } [edit security policies from-zone vpc to-zone db] + /* 1214357 */ + policy allow-mysql-from-nubis-prod { + match { + source-address planb-prod; + destination-address [ bugzilla5.db.scl3 bugzilla6.db.scl3 bugzilla7.db.scl3 bugzilla8.db.scl3 ]; + application mysql; + } + then { + permit; + } + } [edit security zones security-zone db address-book] address bouncer-rw-vip.db.scl3 { ... } + address bugzilla5.db.scl3 10.22.70.100/32; + address bugzilla6.db.scl3 10.22.70.101/32; + address bugzilla7.db.scl3 10.22.70.129/32; + address bugzilla8.db.scl3 10.22.70.102/32; [edit security zones security-zone vpc address-book] address hsm-production-nat2.vpc.scl3 { ... } + address us-east-1-stage-1 10.162.14.96/27; + address us-east-1-stage-2 10.162.14.128/27; + address us-east-1-stage-3 10.162.14.160/27; + address us-west-2-stage-1 10.164.14.96/27; + address us-west-2-stage-2 10.164.14.128/27; + address us-west-2-stage-3 10.164.14.160/27; + address us-east-1-prod-1 10.162.13.96/27; + address us-east-1-prod-2 10.162.13.128/27; + address us-east-1-prod-3 10.162.13.160/27; + address us-west-2-prod-1 10.164.13.96/27; + address us-west-2-prod-2 10.164.13.128/27; + address us-west-2-prod-3 10.164.13.160/27; [edit security zones security-zone vpc address-book] + address-set planb-stage { + address us-east-1-stage-1; + address us-east-1-stage-2; + address us-east-1-stage-3; + address us-west-2-stage-1; + address us-west-2-stage-2; + address us-west-2-stage-3; + } + address-set planb-prod { + address us-east-1-prod-1; + address us-east-1-prod-2; + address us-east-1-prod-3; + address us-west-2-prod-1; + address us-west-2-prod-2; + address us-west-2-prod-3; + }
Status: ASSIGNED → UNCONFIRMED
Ever confirmed: false
Flags: needinfo?(gozer)
Verified working, than you :dcurado! $> nc 10.22.70.129 3306 Host '10.164.13.xxx' is not allowed to connect to this MySQL server Small matter of ACLs on the MySQL side, but that ain't networking!
Flags: needinfo?(gozer)
Flags: needinfo?(scabral)
Granted ACLs for the repl user, ONLY using an x509 certificate, from: us-east-1: ProdPrivateSubnetAZ1Cidr(10.162.13.96/27) ProdPrivateSubnetAZ2Cidr(10.162.13.128/27) ProdPrivateSubnetAZ3Cidr(10.162.13.160/27) us-west-2: ProdPrivateSubnetAZ1Cidr(10.164.13.96/27) ProdPrivateSubnetAZ2Cidr(10.164.13.128/27) ProdPrivateSubnetAZ3Cidr(10.164.13.160/27) We now have: mysql> select user,host,ssl_type from mysql.user where user='repl'; +------+------------------+----------+ | user | host | ssl_type | +------+------------------+----------+ | repl | 10.2.70.55 | X509 | | repl | 10.22.70.4_ | X509 | | repl | 10.8.70.152 | X509 | | repl | 10.8.70.5_ | X509 | | repl | 10.164.13.128/27 | X509 | | repl | 10.164.13.96/27 | X509 | | repl | 10.162.13.96/27 | X509 | | repl | 10.162.13.128/27 | X509 | | repl | 10.162.13.160/27 | X509 | | repl | 10.164.13.160/27 | X509 | +------+------------------+----------+ 10 rows in set (0.00 sec)
username is repl, password will be sent under other cover. If you want to use the same certificates as on bugzilla stage, you can find the certificates in /usr/local/etc/ on bugzilla[12].stage.db.scl3.mozilla.com.
Flags: needinfo?(scabral)
Doesn't seem to be working: ubuntu@ip-10-162-13-125:~$ nc 10.22.70.129 3306 Host '10.162.13.125' is not allowed to connect to this MySQL server And for reference, 10.162.13.125 is part of ProdPrivateSubnetAZ1Cidr(10.162.13.96/27)
a) can you ping between these hosts b) check your security groups c) will nc work if there's some sort of x509 stuff going on?
Flags: needinfo?(gozer)
(In reply to Dave Curado :dcurado from comment #8) > a) can you ping between these hosts [root@ip-10-162-13-175 ec2-user]# ping -c1 10.22.70.102 PING 10.22.70.102 (10.22.70.102) 56(84) bytes of data. 64 bytes from 10.22.70.102: icmp_seq=1 ttl=62 time=76.4 ms > b) check your security groups No restrictions whatesoever > c) will nc work if there's some sort of x509 stuff going on? [root@ip-10-162-13-175 ec2-user]# mysql -vvv --ssl -h10.22.70.102 ERROR 1130 (HY000): Host '10.162.13.175' is not allowed to connect to this MySQL server And I know for a fact that the TCP connection establishes and that error message is coming back from the MySQL server itself, before we can send any data
Flags: needinfo?(gozer)
Here is the policy on the firewall that allows this traffic through: Policy: allow-mysql-from-nubis-prod, action-type: permit, State: enabled, Index: 118, Scope Policy: 0 Policy Type: Configured Sequence number: 1 From zone: vpc, To zone: db Source addresses: us-west-2-prod-3: 10.164.13.160/27 us-west-2-prod-2: 10.164.13.128/27 us-west-2-prod-1: 10.164.13.96/27 us-east-1-prod-3: 10.162.13.160/27 us-east-1-prod-2: 10.162.13.128/27 us-east-1-prod-1: 10.162.13.96/27 Destination addresses: bugzilla8.db.scl3: 10.22.70.102/32 bugzilla7.db.scl3: 10.22.70.129/32 bugzilla6.db.scl3: 10.22.70.101/32 bugzilla5.db.scl3: 10.22.70.100/32 Application: mysql IP protocol: tcp, ALG: 0, Inactivity timeout: 1800 Source port range: [0-0] Destination port range: [3306-3306] Per policy TCP Options: SYN check: No, SEQ check: No It clearly allows this traffic through. As well, I did a search for 10.162.13.175 in splunk, which would see a DENY log from the firewall if there was one... there wasn't one. Because the network is always at fault, until we can prove that it is not, I will add a log statement to the above policy. Try it again, and I'll see if we get a PERMIT log showing your packets came through. That will also allow us to see if your packets are allowed to leave AWS, or if your packets are being rejected by the DBs in SCL3.
Flags: needinfo?(gozer)
(In reply to Dave Curado :dcurado from comment #10) > Here is the policy on the firewall that allows this traffic through: > [...] > It clearly allows this traffic through. 100% agree, my previous comments were more for :sheeri than for you. > As well, I did a search for 10.162.13.175 in splunk, which would see a DENY > log from the firewall if there was one... there wasn't one. > > Because the network is always at fault, until we can prove that it is not, I > will add a log statement to the above policy. Try it again, and I'll see if > we get a PERMIT log showing your packets came through. Sure, but since the error I get is *recieved* back from the DB host I am talking to, that already proves to me that TCP flows both way freely. It's not a network problem for sure. This is an ACL issue on the MySQL side for sure.
Flags: needinfo?(gozer)
MySQL needs "real" netmasks, so 10.162.13.160/27 isn't valid. We changed it to 10.162.13.160/255.255.255.224 and basic connectivity works. We are working on SSL connectivity now.
Assignee: dcurado → scabral
Added MySQL ACLs to the repl user from these hosts: 10.162.13.96/255.255.255.224 10.162.13.128/255.255.255.224 10.162.13.160/255.255.255.224 10.164.13.96/255.255.255.224 10.164.13.128/255.255.255.224 10.164.13.160/255.255.255.224 Did not require any SSL, as they are going over a VPN pipe now. Resolving, I think that's all that was left here.
Status: UNCONFIRMED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.